Using a scene illuminating infrared emitter array in a video monitoring camera for depth determination

ABSTRACT

A method generates depth maps at a camera having illuminators, a lens assembly, an image sensing element, a processor, and memory. The illuminators operate in a first mode to provide illumination, the lens assembly focuses incident light on the image sensing element, the memory stores programs for execution by the processor, and the processor executes the programs to control operation of the camera. The method reconfigures the illuminators to operate in a second mode, where each of a plurality of subsets of the illuminators provides illumination of a scene separately. For each subset, the process activates the illuminators in the subset without activating illuminators not in the subset and receives reflected illumination from the scene incident on the lens assembly and focused onto the image sensing element. The measured light intensity values of the received reflected illumination at the image sensing element are transmitted to a remote server for processing.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/900,766, filed Feb. 20, 2018, entitled “Using a Scene IlluminatingInfrared Emitter Array in a Video Monitoring Camera for DepthDetermination,” which is a continuation of U.S. patent application Ser.No. 14/740,205, filed Jun. 15, 2015, now U.S. Pat. No. 9,900,560, issuedon Feb. 20, 2018, entitled “Using a Scene Illuminating Infrared EmitterArray in a Video Monitoring Camera for Depth Determination,” which is acontinuation of U.S. patent application Ser. No. 14/738,818, filed Jun.12, 2015, now U.S. Pat. No. 9,454,820, issued on Sep. 27, 2016, entitled“Using a Scene Illuminating Infrared Emitter Array in a Video MonitoringCamera for Depth Determination,” each of which is incorporated byreference herein in its entirety.

This application is related to U.S. Provisional Application Ser. No.62/021,620, filed Jul. 7, 2014, entitled “Activity Recognition and VideoFiltering,” which is incorporated by reference herein in its entirety.

This application is related to U.S. patent application Ser. No.14/723,276, filed May 27, 2015, entitled “Multi-Mode LED IlluminationSystem,” which is incorporated by reference herein in its entirety.

This application is related to U.S. patent application Ser. No.14/738,803, filed Jun. 12, 2015, entitled “Simulating an InfraredEmitter Array in a Video Monitoring Camera to Construct a Lookup Tablefor Depth Determination,” which is incorporated by reference herein inits entirety.

This application is related to U.S. patent application Ser. No.14/738,806, filed Jun. 12, 2015, entitled “Using Infrared Images of aMonitored Scene to Identify Windows,” which is incorporated by referenceherein in its entirety.

This application is related to U.S. patent application Ser. No.14/738,817, filed Jun. 12, 2015, entitled “Using a Depth Map of aMonitored Scene to Identify Floors, Walls, and Ceilings,” which isincorporated by reference herein in its entirety.

This application is related to U.S. patent application Ser. No.14/738,825, filed Jun. 12, 2015, entitled “Using Depth Maps of a Sceneto Identify Movement of a Video Camera,” which is incorporated byreference herein in its entirety.

This application is related to U.S. patent application Ser. No.14/738,811, filed Jun. 12, 2015, entitled “Using a Scene IlluminatingInfrared Emitter Array in a Video Monitoring Camera to Estimate thePosition of the Camera,” which is incorporated by reference herein inits entirety.

This application is related to U.S. patent application Ser. No.14/738,816, filed Jun. 12, 2015, entitled “Using a Scene Informationfrom a Security Camera to Reduce False Security Alerts,” which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosed implementations relate generally to video cameras, andmore specifically to using illumination emitters from a video camera toidentify properties of the scene monitored by the camera or to identifyproperties of the camera itself.

BACKGROUND

Video surveillance cameras are used extensively. Usage of video camerasin residential environments has increased substantially, in part due tolower prices and simplicity of deployment. In many cases, surveillancecameras include infrared emitters in order to illuminate a scene whenlight from other sources is limited or absent.

Some video cameras enable a user to identify “zones” within the scenethat is visible to the camera. This can be useful to identify movementor changes within those zones.

Because a surveillance camera can capture a very large amount of data(e.g., running 24 hours a day, 7 days a week), some cameras enable auser to set up alerts based on specific criteria. The criteria caninclude movement within a scene, movement of a specific type, ormovement within a certain time range.

SUMMARY

Accordingly, there is a need for camera systems that provide simplerusage and better utilization. In various implementations, the disclosedfunctionality complements or replaces the functionality of existingcamera systems.

In accordance with some implementations, a process generates lookuptables for use in estimating spatial depth in a visual scene. Theprocess is performed at a server having one or more processors andmemory. The memory stores one or more programs configured for executionby the one or more processors. The process identifies a plurality ofdistinct subsets of IR illuminators of a camera system. The camerasystem has a 2-dimensional array of image sensors (e.g., photodiodes)and a plurality of IR illuminators in fixed locations relative to thearray of image sensors. The process partitions the image sensors into aplurality of pixels. In some implementations, each pixel comprises asingle image sensor. In some implementations, each pixel comprises aplurality of image sensors, which can be 50 or more. For each pixel andfor each of m distinct depths from the respective pixel, the processsimulates a virtual surface at the respective depth. In someimplementations, the simulated virtual surfaces are planar, but in otherimplementations the simulated surfaces are spherical, parabolic, orcubic. For each of the distinct subsets of IR illuminators, the processdetermines an expected IR light intensity at the respective pixel basedon the respective depth and based on only the respective subset of IRilluminators emitting IR light. The process then forms an intensityvector using the expected IR light intensities for each of the distinctsubsets, and normalizes the intensity vector. For each pixel, theprocess constructs a lookup table comprising the normalized vectorscorresponding to the pixel. The lookup table associates each respectivenormalized vector with the respective depth of the respective simulatedsurface.

In some implementations, the expected IR light intensity at therespective pixel is based on characteristics of the IR illuminators ofthe camera system. In some implementations, the characteristics includelux, orientation of the IR illuminators relative to the sensor array,and/or location of the IR illuminators relative to the sensor array.

In some implementations, the process normalizes each intensity vector bycomputing a respective magnitude of the intensity vector and dividingeach component of the intensity vector by the respective magnitude.

In some implementations, the array of image sensors comprises more thanone million image sensors. In some implementations, the array of imagesensors is downsampled to a smaller number of pixels. For example, anarray of image sensors with one million individual sensors may bedownsampled to 10,000 pixels. The downsampling used (if any) may dependon available resources, such as memory, bandwidth, processor speed,and/or number of processors.

In accordance with some implementations, a process creates a depth mapof a scene. The process is performed at a computing device having one ormore processors and memory. The memory stores one or more programsconfigured for execution by the one or more processors. For each of aplurality of distinct subsets of IR illuminators of a camera system, theprocess receives a captured IR image of a first scene taken by a2-dimensional array of image sensors of the camera system while therespective subset of IR illuminators are emitting IR light and the IRilluminators not in the respective subset are not emitting IR light. Theimage sensors are partitioned into a plurality of pixels. In someimplementations, each pixel comprises a single image sensor, but inother implementations, each pixel comprises a plurality of imagesensors. In some implementations, the computing device is a server, andthe captured images are received from a remotely located camera. In someimplementations, the computing device is included in a camera, and theimages are processed locally at the camera. For each pixel of theplurality of pixels, the process uses the captured IR images to form arespective vector of light intensity at the respective pixel. Theprocess then estimates a depth in the first scene at the respectivepixel by looking up the respective vector in a respective lookup table.In some implementations, the lookup table is stored at the camera systemduring a calibration process.

In some implementations, looking up the respective vector in therespective lookup table includes computing an inner product of therespective vector with records in the lookup table. In someimplementations, the inner product is computed for each record in thelookup table. The process computes the depth in the first scene at thepixel as a depth corresponding to a record in the lookup table whoseinner product with the respective vector is greatest among the computedinner products for the respective vector.

In some implementations, each respective vector for a respective pixelcomprises a plurality of components, with each of the componentscorresponding to a respective IR light intensity for the respectivepixel for a respective captured IR image. In some implementations,computing an inner product comprises computing a dot product.

In some implementations, the IR illuminators are orientated at aplurality of distinct angles relative to the array of image sensors.

In some implementations, the depth map of the first scene is created inresponse to detecting a trigger event. In some implementations, thetrigger event is detecting movement of a first object in the first scenefrom a first location to a second location. In some implementations, thetrigger event is a power interruption event.

In some implementations, a respective lookup table is generated duringthe calibration process. In some implementations, the calibrationprocess includes simulating a virtual planar surface at a plurality ofrespective depths in the first scene and determining, for each pixel andeach respective depth, an expected IR light intensity.

Implementations select the distinct subsets of IR illuminators invarious ways. In some implementations, each of the distinct subsets ofIR illuminators comprises two adjacent IR illuminators, and the distinctsubsets of IR illuminators are non-overlapping.

In some implementations, each respective lookup table includes aplurality of normalized IR light intensity vectors, and each normalizedlight intensity vector corresponds to a respective depth in the firstscene.

In some implementations, the respective lookup tables are downloaded tothe camera system from a remote server during an initialization processprior to creating the depth map.

In some implementations, prior to capturing the IR images, the processswitches from a first mode of the camera system to a second mode of thecamera system, including deactivating the first mode and activating thesecond mode. In some implementations, the array of image sensors has anassociated first pixel gain curve while the first mode is activated, andthe array of image sensors has an associated second pixel gain curvewhile the second mode is activated.

In some implementations, the process receives a baseline IR image of thescene captured by the array of sensors while none of the IR illuminatorsare emitting IR light. Then, forming each respective vector of lightintensity at a respective pixel comprises subtracting a light intensityat the pixel of the baseline IR image from the light intensity at thepixel of each of the captured IR images.

In accordance with some implementations, a process classifies objects ina scene. The process is performed at a computing device having one ormore processors and memory. The memory stores one or more programsconfigured for execution by the one or more processors. In someimplementations, the computing device is included in a camera system. Insome implementations, the computing device is a server distinct from thecamera system. The process receives a captured IR image of a scene takenby a 2-dimensional image sensor array of the camera system while one ormore IR illuminators of the camera system are emitting IR light. In thisway, the process forms an IR intensity map of the scene with arespective intensity value determined for each pixel of the IR image.The process uses the IR intensity map to identify a plurality of pixelswhose corresponding intensity values are within a predefined intensityrange (e.g., all intensity values between 0 and a positive finite valueor all values between two positive finite values). The process thenclusters the identified plurality of pixels into one or more regionsthat are substantially contiguous. The process determines that a firstregion of the one or more regions corresponds to a specific materialbased, at least in part, on the intensity values of the pixels in thefirst region, and stores information in the memory that identifies thefirst region.

In some implementations, each pixel of the IR image corresponds to aunique respective image sensor in the image sensor array. In someimplementations, the pixels of the IR image form a partition of theimage sensors in the image sensor array and at least one pixelcorresponds to a plurality of image sensors in the image sensor array.

In some implementations, the camera system has a plurality of IRilluminators, and forming an IR intensity map of the scene includesreceiving a respective IR sub-image of the scene for each of a pluralityof distinct subsets of IR illuminators. Each IR sub-image is capturedwhile the respective subset of IR illuminators are emitting IR light andthe IR illuminators not in the respective subset are not emitting IRlight. The respective intensity value for a respective pixel is theaverage of intensity values at the pixel in each of the sub-images.

In some implementations, clustering the identified plurality of pixelsinto one or more regions further comprises using a depth map that wasconstructed using the image sensor array.

In some implementations, clustering the identified plurality of pixelsinto one or more regions further comprises using an RGB image of thescene captured using the image sensor array.

In some implementations, determining that a first region of the one ormore regions corresponds to a specific material comprises determiningthat the first region is substantially a quadrilateral. In someimplementations, the first region is substantially a quadrilateral whena total absolute difference in area between the first region and thequadrilateral is less than a threshold percentage of the quadrilateral'sarea (e.g., 5%, 10%, or 20%).

In some implementations, the predefined intensity range includes allintensity values below a threshold value, and the specific material isglass. The process thereby determines that the first region correspondsto a window in the scene.

In some implementations, the process receives a video stream of thescene from the camera system and reviews the video stream to detectmovement in the scene. The first region is excluded from movementdetection. The process generates a motion alert when there is motiondetected at the scene outside of the first region.

In accordance with some implementations, a process identifies largeplanar surfaces in scenes, such as floors, walls, and ceilings. Theprocess is performed at a computing device having one or more processorsand memory. The memory stores one or more programs configured forexecution by the one or more processors. The process receives aplurality of captured IR images of a scene taken by a 2-dimensionalarray of image sensors of a camera system. Each IR image is capturedwhen a distinct subset of IR illuminators of the camera system areilluminated. The process constructs a depth map of a scene using theplurality of IR images, and uses the depth map to compute a binary depthedge map for the scene. The binary depth edge map identifies whichpoints in the depth map comprise depth discontinuities. The processidentifies a plurality of contiguous components based on the binarydepth edge map. The process determines that a first component of theplurality of contiguous components represents a large planar surface inthe scene by fitting a plane to points in the first component,determining the orientation of the plane, and determining that the planefitting residual error is less than a predefined threshold.

In some implementations, the nature of the large plane is determined byits orientation. When the orientation of the plane is upwards, the planeis determined to be a floor. When the orientation of the plane isdownwards, the plane is determined to be a ceiling. And when theorientation of the plane is horizontal, the plane is determined to be awall.

In some implementations, the computing device is a server distinct fromthe camera system. In other implementations, the computing device isincluded in the camera system.

In some implementations, the image sensors are partitioned into aplurality of pixels. For each pixel, the process uses the captured IRimages to form a respective vector of light intensity at the respectivepixel and estimates a depth in the first scene at the respective pixelusing the respective vector and a respective lookup table. In this way,the process constructs the depth map.

In accordance with some implementations, a process recomputes zones fora scene. The process is performed at a computing device that has one ormore processors and memory. The memory stores one or more programsconfigured for execution by the one or more processors. The processreceives a first RGB image of a scene taken by a 2-dimensional array ofimage sensors of a camera system at a first time. The process alsoreceives a first plurality of distinct IR images of the scene taken bythe array of image sensors temporally proximate to the first time. Eachof the IR images is taken while a different subset of IR illuminators ofthe camera system is emitting light. Using the first plurality of IRimages, the process constructs a first depth map of the scene. The firstdepth map indicates a respective depth in the scene at a plurality ofpixels, where each pixel corresponds to one or more of the imagesensors. The process receives designation from a user of a zone withinthe first RGB image. The zone corresponds to a contiguous plurality ofpixels. At a second time later, the process receives a second pluralityof distinct IR images of the scene taken by the array of image sensors.Each of the IR images in the second plurality is taken while a differentsubset of IR illuminators of the camera system is emitting light. Usingthe second plurality of IR images, the process constructs a second depthmap of the scene. The process then determines physical movement of thecamera system based on the first and second depth maps. Based on thedetermined physical movement, the process translates the zone in thefirst RGB image into an adjusted zone.

In some instances, the determined physical movement is an angularrotation. In some instances, the determined physical movement is alateral displacement. In some instances, the determined physicalmovement includes both an angular rotation and a lateral displacement.Lateral displacements are commonly horizontal, but they can be verticalas well. As used herein, a lateral displacement is any movement in whichthe camera continues to point in the same direction. This includes anycombination of left/right, up/down, and/or forward/backward.

In some implementations, determining the physical movement of the camerasystem includes identifying a plurality of points in the first depth mapand a corresponding plurality of points in the second depth map and theprocess determines a respective displacement for each of the pointsbetween the first and second depth maps.

In some instances, the zone is a first quadrilateral. In some instances,the adjusted zone is a second quadrilateral, and a first edge of thefirst quadrilateral has a length that is different from a correspondingsecond edge of the second quadrilateral.

In some implementations, the process creates the first depth map of thescene by partitioning the image sensors into a plurality of pixels. Foreach pixel, the process forms a respective vector of the received IRimages at the respective pixel and estimates a depth in the scene at therespective pixel by looking up the respective vector in a respectivelookup table.

In some implementations, the computing device is a server distinct fromthe camera system. In other implementations, the computing device isincluded in the camera system.

In some implementations, the process receives a second RGB image of thescene taken by the image sensor array of the camera system temporallyproximate to the second time and correlates the adjusted zone to a setof pixels from the second RGB image.

In some implementations, the process determines the physical movement ofthe camera system using point clouds. The process forms a first pointcloud using a first plurality of points from the first depth map andforms a second point cloud using a second plurality of points from thesecond depth map. The process then computes a minimal transformationthat aligns the first point cloud with the second point cloud. Thisprocess is referred to as “registration.”

In accordance with some implementations, a process estimates the heightand tilt angle of a camera system. The camera system has a 2-dimensionalarray of image sensors and a plurality of IR illuminators in fixedlocations relative to the array of image sensors. The process isperformed at a computing device having one or more processors andmemory. The memory stores one or more programs configured for executionby the one or more processors. In some implementations, the computingdevice is included in the camera system. In some implementations, thecomputing device is a server distinct from the camera system. Theprocess identifies a plurality of distinct subsets of the IRilluminators. In some implementations, each of the distinct subsets ofthe IR illuminators comprises two adjacent IR illuminators, and thedistinct subsets of the IR illuminators are non-overlapping. In someimplementations, one or more of the subsets of IR illuminators comprisesa single IR illuminator. The process partitions the image sensors into aplurality of pixels. In some implementations, each pixel corresponds toa single image sensor. In some implementations, some of the pixelscorrespond to multiple image sensors (e.g., by downsampling).

In accordance with some implementations, for each of a plurality ofheights and tilt angles, the process constructs a dictionary entry thatcorresponds to the camera system having the respective height and tiltangle above a floor. The respective dictionary entry includes respectiveIR light intensity values for pixels in images corresponding toactivating individually each of the distinct subsets of the IRilluminators.

In some implementations, the constructed dictionary entries are based onsimulating the camera, the floor, and the images, and computing expectedIR light intensity values for pixels in the simulated images. In someimplementations, each expected IR light intensity value is based oncharacteristics of the IR illuminators, including one or morecharacteristics selected from the group consisting of lux, orientationof the IR illuminators relative to the array of image sensors, andlocation of the IR illuminators relative to the array of image sensors.In some implementations, a respective dictionary entry for a respectiveheight and respective tilt angle is based on measuring IR lightintensity values of actual images captured by the camera having therespective height and respective tilt angle with respect to an actualfloor.

In accordance with some implementations, for each of the plurality ofdistinct subsets of the IR illuminators, the process receives a capturedIR image of a scene taken by the array of image sensors while therespective subset of the IR illuminators are emitting IR light and theIR illuminators not in the respective subset are not emitting IR light.Using at least one of the captured IR images, the process identifies afloor region corresponding to a floor in the scene. In someimplementations, identifying the floor region includes constructing adepth map of the scene using the captured IR images, identifying aregion bounded by depth discontinuities, and determining that the regionis substantially planar and facing upwards.

In accordance with some implementations, the process forms a vector(sometimes referred to as a feature vector) including pixels from thecaptured IR images in the identified floor region and estimates thecamera height and camera tilt angle relative to the floor by comparingthe feature vector to the dictionary entries.

In some implementations, the respective expected IR light intensity isbased on characteristics of the IR illuminators. In someimplementations, these characteristics include one or more of:illuminator lux; orientation of the IR illuminators relative to thearray of image sensors; and location of the IR illuminators relative tothe array of image sensors.

In some implementations, constructing a dictionary entry includesnormalizing the dictionary entry. In some implementations, normalizing adictionary entry includes determining a respective total magnitude ofthe light intensity features in the dictionary entry and dividing eachcomponent of the dictionary entry by the respective total magnitude. Insome implementations, the dictionary entries are downloaded to thecamera system from the computing device during an initializationprocess.

In some implementations, the process receives a baseline IR image of thescene captured by the array of image sensors while none of the IRilluminators are emitting IR light and subtracts the light intensity ateach pixel of the baseline IR image from the light intensity at thecorresponding pixel of each of the other captured IR images.

In some implementations, estimating the camera height and camera tiltangle relative to the floor includes computing a respective distancebetween the feature vector and respective dictionary entries. Theprocess selects a first dictionary entry whose corresponding computeddistance is less than the other computed distances and estimates thecamera height and tilt angle to be the height and tilt angle associatedwith the first dictionary entry. In some implementations, computing arespective distance between the feature vector and respective dictionaryentries comprises computing a Euclidean distance that uses only vectorcomponents corresponding to pixels in the identified floor region. Insome implementations, the process normalizes the feature vector and thedictionary entries prior to computing the distances.

In accordance with some implementations, a process reduces falsepositive security alerts. The process is performed at a computing devicehaving one or more processors, and memory storing one or more programsconfigured for execution by the one or more processors. In someimplementations, the computing device is a server distinct from a videocamera. In some implementations, the computing device is included in thevideo camera. The process computes a depth map for a scene monitored bya video camera using a plurality of IR images captured by the videocamera and uses the depth map to identify a first region within thescene having historically above average false positive detected motionevents. The process monitors a video stream provided by the video camerato identify motion events. The monitored area excludes the first region.The process generates a motion alert when there is detected motion inthe scene outside of the first region and the detected motion satisfiesthreshold criteria. In some implementations, satisfying the thresholdcriteria includes detecting movement of an object in the scene, and thedetected movement exceeds a predefined distance within a predefinedperiod of time. In some implementations, satisfying the thresholdcriteria includes detecting movement for an object that exceeds apredefined size. In some implementations, satisfying the thresholdcriteria includes detecting simultaneous movement of two or more objectsin the scene.

In some implementations, the video camera has a plurality of IRilluminators and each of the plurality of IR images captured by thevideo camera is taken when a different subset of the illuminators isemitting light.

In some instances, the first region is identified as a ceiling. In someimplementations, identifying the first region as a ceiling includesusing the depth map to compute a binary depth edge map for the scene.The binary depth edge map identifies which points in the depth mapcomprise depth discontinuities. In some implementations, identifying thefirst region as a ceiling also includes identifying a contiguouscomponent based on the binary depth edge map. In some implementations,identifying the first region as a ceiling also includes fitting a planeto points in the contiguous component, determining that the planefitting residual error is less than a predefined threshold, anddetermining that the plane is oriented downward.

In some instances, the first region is identified as a window. In someimplementations, identifying the first region as a window includesidentifying the first region as a region of low light intensity within acaptured IR image of the scene, fitting the first region with aquadrilateral, and determining that the absolute difference between thefirst region and the quadrilateral is less than a threshold percentageof the area of the quadrilateral.

In some instances, the first region is identified as a television.

In accordance with some implementations, process for generating depthmaps is performed by a camera having a plurality of illuminators, a lensassembly, an image sensing element, a processor, and memory. Theilluminators are configured to operate in a first mode to provideillumination using all of the illuminators, the lens assembly isconfigured to focus incident light on the image sensing element, thememory is configured to store image data from the image sensing element,and the processor is configured to execute programs to control operationof the camera. The process reconfigures the plurality of illuminators tooperate in a second mode, where each of a plurality of subsets of theplurality of illuminators provides illumination separately from othersubsets of the plurality of illuminators. The process sequentiallyactivates each of the subsets of the illuminators to illuminate a sceneand receives reflected illumination from the illuminated scene incidenton the lens assembly and focused onto the image sensing element. Theprocess measures light intensity values of the received reflectedillumination at the image sensing element and stores to the memory themeasured light intensity values associated with activation of each ofthe subsets.

In some implementations, each of the subsets of illuminators isconfigured at a different angle relative to the image sensing element.

In some implementations, each of the subsets of illuminators highlightsa different portion of the scene.

In some implementations, the process transmits the stored lightintensity values to a depth mapping module configured to estimatespatial depths of objects in the scene based on the stored lightintensity values, predetermined illumination specifications of theilluminators, and response specifications of the image sensors.

In some implementations, the illuminators are IR illuminators.

In some implementations, the illuminators comprise 8 IR illuminators andeach of the subsets of the illuminators comprises 2 adjacent IRilluminators.

In some implementations, the image sensing element is a 2-dimensionalarray of image sensors.

In some implementations, differences in the stored light intensityvalues associated with activation of each of the subsets for arespective image sensor correlate with spatial depth of an object in thescene from which reflected light was received at the respective imagesensor.

In some implementations, the process captures a baseline image whilenone of the illuminators are emitting light. The captured baseline imagemeasures ambient light intensity of the scene at each of the imagesensors. The process stores the captured baseline image to the memoryand for each image sensor, the process subtracts the baseline intensityvalue from the stored intensity values for the respective image sensorto correct the stored intensity values for ambient light at the scene.

In some implementations, the image sensors are partitioned into aplurality of pixels and for each pixel of the plurality of pixels theprocess using the captured IR images to form a respective vector oflight intensity at the respective pixel. For each pixel, the processalso estimates a depth in the first scene at the respective pixel bylooking up the respective vector in a respective lookup table. In someimplementations, looking up the respective vector in the respectivelookup table includes computing an inner product of the respectivevector with records in the lookup table and determining the depth in thefirst scene at the pixel as a depth corresponding to a record in thelookup table whose inner product with the respective vector is greatestamong the computed inner products for the respective vector. In someimplementations, computing an inner product of the respective vectorwith records in the lookup table includes computing an inner product ofthe respective vector and the respective record for each record in therespective lookup table. In some implementations, the respective vectorfor a respective pixel has a plurality of components, each of thecomponents corresponds to a respective IR light intensity for therespective pixel for a respective captured IR image, and computing aninner product comprises computing a dot product.

In some implementations, each respective lookup table includes aplurality of normalized IR light intensity vectors, each normalizedlight intensity vector corresponds to a respective depth in the firstscene.

In some implementations, the respective lookup table is downloaded tothe camera system from a remote server during an initialization process.

In accordance with some implementations, a computing device has one ormore processors, memory, and one or more programs stored in the memory.The programs are configured for execution by the one or more processors.The one or more programs including instructions for performing any ofthe processes described herein. In some implementations, the computingdevice is a server, which is distinct from a camera system. In otherimplementations, the computing device includes a camera.

In accordance with some implementations, a non-transitory computerreadable storage medium stores one or more programs configured forexecution by a computing device having one or more processors andmemory. The one or more programs include instructions for performing anyof the processes described herein. In some implementations, thecomputing device is a server, which is distinct from a camera system. Inother implementations, the computing device includes a camera.

Thus, computing devices, server systems, and camera systems are providedwith more efficient methods for utilizing IR emitters and a sensor arrayto classify objects in a scene or simplify creation of alerts. Thesedisclosed camera systems thereby increase the effectiveness, efficiency,and user satisfaction with such systems. Such methods may complement orreplace conventional methods.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described implementations,reference should be made to the Description of Implementations below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is a representative smart home environment in accordance withsome implementations.

FIG. 2 is a block diagram illustrating a representative networkarchitecture that includes a smart home network in accordance with someimplementations.

FIG. 3 illustrates a network-level view of an extensible platform fordevices and services, which may be integrated with the smart homeenvironment of FIG. 1 in accordance with some implementations.

FIG. 4 illustrates an abstracted functional view of the extensibleplatform of FIG. 3, with reference to a processing engine as well asdevices of the smart home environment, in accordance with someimplementations.

FIG. 5 is a representative operating environment in which a video serversystem interacts with client devices and video sources in accordancewith some implementations.

FIG. 6 is a block diagram illustrating a representative video serversystem in accordance with some implementations.

FIG. 7 is a block diagram illustrating a representative client device inaccordance with some implementations.

FIG. 8 is a block diagram illustrating a representative video capturingdevice (e.g., a camera) in accordance with some implementations.

FIG. 9 is a block diagram of a scene understanding server in accordancewith some implementations.

FIGS. 10-12 illustrate the illuminators and array of memory sensors fora camera in accordance with some implementations.

FIGS. 13, 14, 15A, and 15B illustrate a process of building a lookuptable for depth estimation in accordance with some implementations.

FIGS. 16A-16D, 17A, and 17B illustrate a process of creating a depth mapusing a sequence of captured IR images in accordance with someimplementations.

FIGS. 18A-18E illustrate a process for identifying objects in a scenebased on specularity, in accordance with some implementations.

FIGS. 19A-19I illustrate a process of zone recalculation in accordancewith some implementations.

FIGS. 20A-20K illustrate a process of identifying floors, walls, andceilings in a scene in accordance with some implementations.

FIGS. 21A-21E illustrate a process of estimating camera pose inaccordance with some implementations.

FIGS. 22A-22C provide a flowchart of a process for building a lookuptable in accordance with some implementations.

FIGS. 23A-23C provide a flowchart of a process for using a lookup tableto build a depth map of a scene in accordance with some implementations.

FIGS. 24A-24C provide a flowchart of a process for identifying objects,such as windows, based on specularity, in accordance with someimplementations.

FIGS. 25A-25B provide a flowchart of a process for identifying floors,walls, ceilings, and other large planar surfaces in accordance with someimplementations.

FIGS. 26A-26C provide a flowchart of a process for correcting useridentified zones when a camera is moved according to someimplementations.

FIGS. 27A-27D provide a flowchart of a process for estimating camerapose in accordance with some implementations.

FIGS. 28-30 provide an overview of some of the processes described, andprovide an overview of how the processes work together according to someimplementations.

FIGS. 31A-31E illustrate how some implementations address movement of acamera.

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DESCRIPTION OF IMPLEMENTATIONS

Security cameras typically include illuminators so that video capture ispossible even in low light conditions or in complete darkness. Many suchcameras use infrared (IR) illuminators, which allow video capturewithout illuminating a scene with visible light. Typically, whenillumination is needed, all of the illuminators are turned on.

Disclosed implementations utilize existing illuminators in differentways so that the camera can provide more information about a scene. Onestep in some implementations is to control the illuminators individuallyor in small groups rather than turning them all on or off together.Because the illuminators are in different locations with respect to theimage sensor array, captured images are slightly different depending onwhich illuminators are on, as illustrated below in FIGS. 16A-16D.

As described below, some implementations build a depth map of a sceneusing the differences in captured images when different illuminators areon. A depth map estimates the distance between the image sensor array ofthe camera and the nearest object for each pixel in the field of visionof the camera. In some implementations, the depth map is implemented asan m×n matrix of depths, where m×n is the arrangement of pixelscorresponding to image sensor array.

In some implementations, there is a one-to-one correspondence betweenpixels and individual image sensors in the array, but in manyimplementations the images are downsampled to create a more manageableset of pixels (e.g., 10,000 pixels instead of 1,000,000 pixels).

A depth map can be used in various ways to determine information about ascene. In some implementations, the depth map is used to help identifyfloors, walls, and ceilings. In some implementations, the depth maphelps to identify when a camera has moved slightly, enabling automaticzone correction for previously defined zones in the scene. In someimplementations, the depth map helps to identify the position of thecamera (e.g., height above the floor and angle). These features provideuseful information, and also allow for more accurate alerts. Forexample, if a region is identified as a ceiling, perceiving “movement”in that region is likely to be light reflections instead of an intruder.As another example, automatic zone correction can ensure that the properregion is monitored (e.g., a doorway) even if the zone is in a differentlocation relative to a new camera position (e.g., because the camera wasbumped).

Some implementations also enable detection of windows usingcharacteristics of windows that are different from other objects. Forexample, whereas light incident on most objects scatters in alldirections, light incident on a window either passes through the windowor reflects off like a mirror. Identifying windows can be useful invarious ways, including the prevention of false alerts. For example,movement of leaves on a tree outside of a window does not constitute anintruder inside a monitored room with the window.

These features may be implemented for an independent camera, but in someimplementations, the camera is part of a smart home environment 100, asdescribed below in FIGS. 1-8.

Video-based surveillance and security monitoring of a premises generatesa continuous video feed that may last hours, days, and even months.Although motion-based recording triggers can help trim down the amountof video data that is actually recorded, there are a number of drawbacksassociated with video recording triggers based on simple motiondetection in the live video feed. For example, when motion detection isused as a trigger for recording a video segment, the threshold of motiondetection must be set appropriately for the scene of the video;otherwise, the recorded video may include many video segments containingtrivial movements (e.g., lighting change, leaves moving in the wind,shifting of shadows due to changes in sunlight exposure, etc.) that areof no significance to a reviewer. On the other hand, if the motiondetection threshold is set too high, video data on important movementsthat are too small to trigger the recording may be irreversibly lost.Furthermore, at a location with many routine movements (e.g., carspassing through in front of a window) or constant movements (e.g., ascene with a running fountain, a river, etc.), recording triggers basedon motion detection are rendered ineffective, because motion detectioncan no longer accurately select out portions of the live video feed thatare of special significance. As a result, a human reviewer has to siftthrough a large amount of recorded video data to identify a small numberof motion events after rejecting a large number of routine movements,trivial movements, and movements that are of no interest for a presentpurpose.

Due to at least the challenges described above, it is desirable to havea method that maintains a continuous recording of a live video feed suchthat irreversible loss of video data is avoided and, at the same time,augments simple motion detection with false positive suppression andmotion event categorization. The false positive suppression techniqueshelp to downgrade motion events associated with trivial movements andconstant movements. The motion event categorization techniques help tocreate category-based filters for selecting only the types of motionevents that are of interest for a present purpose. As a result, thereviewing burden on the reviewer may be reduced. In addition, as thepresent purpose of the reviewer changes in the future, the reviewer cansimply choose to review other types of motion events by selecting theappropriate motion categories as event filters.

In addition, in some implementations, event categories can also be usedas filters for real-time notifications and alerts. For example, when anew motion event is detected in a live video feed, the new motion eventis immediately categorized, and if the event category of the newlydetected mention event is a category of interest selected by a reviewer,a real-time notification or alert can be sent to the reviewer regardingthe newly detected motion event. In addition, if the new event isdetected in the live video feed as the reviewer is viewing a timeline ofthe video feed, the event indicator and the notification of the newevent will have an appearance or display characteristic associated withthe event category.

Furthermore, the types of motion events occurring at different locationsand settings can vary greatly, and there are many event categories forall motion events collected at the video server system (e.g., the videoserver system 508). Therefore, it may be undesirable to have a set offixed event categories from the outset to categorize motion eventsdetected in all video feeds from all camera locations for all users. Insome implementations, the motion event categories for the video streamfrom each camera are gradually established through machine learning, andare thus tailored to the particular setting and use of the video camera.

In addition, in some implementations, as new event categories aregradually discovered based on clustering of past motion events, theevent indicators for the past events in a newly discovered eventcategory are refreshed to reflect the newly discovered event category.In some implementations, a clustering algorithm automatically phases outold, inactive, and/or sparse categories when categorizing motion events.As a camera changes location, event categories that are no longer activeare gradually retired without manual input to keep the motion eventcategorization model current. In some implementations, user input toedit the assignment of past motion events into respective eventcategories is also taken into account for future event categoryassignment and new category creation.

In some circumstances, there are multiple objects moving simultaneouslywithin the scene of a video feed. In some implementations, the motiontrack associated with each moving object corresponds to a respectivemotion event candidate, such that the movement of the different objectsin the same scene may be assigned to different motion event categories.

In general, motion events may occur in different regions of a scene atdifferent times. Out of all the motion events detected within a scene ofa video stream over time, a reviewer may only be interested in motionevents that occur within or enter a particular zone of interest in thescene. In addition, the zones of interest may not be known to thereviewer and/or the video server system until long after one or moremotion events of interest have occurred within the zones of interest.For example, a parent may not be interested in activities centeredaround a cookie jar until after some cookies have mysteriouslydisappeared. Furthermore, the zones of interest in the scene of a videofeed can vary for a reviewer over time depending on the present purposeof the reviewer. For example, the parent may be interested in seeing allactivities that occurred around the cookie jar one day when some cookiesare missing, and the parent may be interested in seeing all activitiesthat occurred around a mailbox the next day when some expected mail ismissing. Accordingly, in some implementations, the techniques disclosedherein allow a reviewer to define and create one or more zones ofinterest within a static scene of a video feed, and then use the createdzones of interest to retroactively identify all past motion events (orall motion events within a particular past time window) that havetouched or entered the zones of interest. In some implementations, theidentified motion events are presented to the user in a timeline or in alist. In some implementations, real-time alerts for any new motionevents that touch or enter the zones of interest are sent to thereviewer. The ability to quickly identify and retrieve past motionevents that are associated with a newly created zone of interestaddresses the drawbacks of conventional zone monitoring techniques.Conventionally, the zones of interest must be defined first based on acertain degree of guessing and anticipation that may later prove to beinadequate or wrong. Also, in conventional systems, only future events(as opposed to both past and future events) within the zones of interestcan be identified.

In some implementations, when detecting new motion events that havetouched or entered some zone(s) of interest, the event detection isbased on the motion information collected from the entire scene, ratherthan just within the zone(s) of interest. In particular, aspects ofmotion detection, motion object definition, motion track identification,false positive suppression, and event categorization are all based onimage information collected from the entire scene, rather than justwithin each zone of interest. As a result, context around the zones ofinterest is taken into account when monitoring events within the zonesof interest. Thus, the accuracy of event detection and categorizationmay be improved as compared to conventional zone monitoring techniquesthat perform all calculations with image data collected only within thezones of interest.

FIGS. 1-4 provide an overview of exemplary smart home device networksand capabilities. FIGS. 5-8 provide a description of the systems anddevices participating in the video monitoring.

Reference will now be made in detail to implementations, examples ofwhich are illustrated in the accompanying drawings. In the followingdetailed description, numerous specific details are set forth in orderto provide a thorough understanding of the various describedimplementations. However, it will be apparent to one of ordinary skillin the art that the various described implementations may be practicedwithout these specific details. In other instances, well-known methods,procedures, components, circuits, and networks have not been describedin detail so as not to unnecessarily obscure aspects of theimplementations.

FIG. 1 depicts a representative smart home environment in accordancewith some implementations. The smart home environment 100 includes astructure 150, which may be a house, office building, garage, or mobilehome. It will be appreciated that devices may also be integrated into asmart home environment 100 that does not include an entire structure150, such as an apartment, condominium, or office space. Further, thesmart home environment may control and/or be coupled to devices outsideof the actual structure 150. Indeed, several devices in the smart homeenvironment need not be physically within the structure 150. Forexample, a device controlling a pool heater 114 or irrigation system 116may be located outside of structure 150.

The depicted structure 150 includes a plurality of rooms 152, separatedat least partly from each other via walls 154. The walls 154 may includeinterior walls or exterior walls. Each room may further include a floor156 and a ceiling 158. Devices may be mounted on, integrated with,and/or supported by a wall 154, a floor 156, or a ceiling 158.

In some implementations, the smart home environment 100 includes aplurality of devices, including intelligent, multi-sensing,network-connected devices, that integrate seamlessly with each other ina smart home network 202 and/or with a central server or acloud-computing system to provide a variety of useful smart homefunctions. The smart home environment 100 may include one or moreintelligent, multi-sensing, network-connected thermostats 102 (“smartthermostats”), one or more intelligent, network-connected, multi-sensinghazard detection units 104 (“smart hazard detectors”), and one or moreintelligent, multi-sensing, network-connected entryway interface devices106 (“smart doorbells”). In some implementations, the smart thermostat102 detects ambient climate characteristics (e.g., temperature and/orhumidity) and controls a HVAC system 103 accordingly. The smart hazarddetector 104 may detect the presence of a hazardous substance or asubstance indicative of a hazardous substance (e.g., smoke, fire, and/orcarbon monoxide). The smart doorbell 106 may detect a person's approachto or departure from a location (e.g., an outer door), control doorbellfunctionality, announce a person's approach or departure via audio orvisual means, and/or control settings on a security system (e.g., toactivate or deactivate the security system when occupants go and come).

In some implementations, the smart home environment 100 includes one ormore intelligent, multi-sensing, network-connected wall switches 108(“smart wall switches”), along with one or more intelligent,multi-sensing, network-connected wall plug interfaces 110 (“smart wallplugs”). The smart wall switches 108 may detect ambient lightingconditions, detect room-occupancy states, and control a power and/or dimstate of one or more lights. In some instances, smart wall switches 108may also control a power state or speed of a fan, such as a ceiling fan.The smart wall plugs 110 may detect occupancy of a room or enclosure andcontrol supply of power to one or more wall plugs (e.g., such that poweris not supplied to the plug if nobody is at home).

In some implementations, the smart home environment 100 includes aplurality of intelligent, multi-sensing, network-connected appliances112 (“smart appliances”), such as refrigerators, stoves, ovens,televisions, washers, dryers, lights, stereos, intercom systems,garage-door openers, floor fans, ceiling fans, wall air conditioners,pool heaters, irrigation systems, security systems, space heaters,window AC units, motorized duct vents, and so forth. In someimplementations, when plugged in, an appliance may announce itself tothe smart home network, such as by indicating what type of appliance itis, and it may automatically integrate with the controls of the smarthome. Such communication by the appliance to the smart home may befacilitated by either a wired or wireless communication protocol. Thesmart home may also include a variety of non-communicating legacyappliances 140, such as old conventional washer/dryers, refrigerators,and the like, which may be controlled by smart wall plugs 110. The smarthome environment 100 may further include a variety of partiallycommunicating legacy appliances 142, such as infrared (“IR”) controlledwall air conditioners or other IR-controlled devices, which may becontrolled by IR signals provided by the smart hazard detectors 104 orthe smart wall switches 108.

In some implementations, the smart home environment 100 includes one ormore network-connected cameras 118 that are configured to provide videomonitoring and security in the smart home environment 100.

The smart home environment 100 may also include communication withdevices outside of the physical home but within a proximate geographicalrange of the home. For example, the smart home environment 100 mayinclude a pool heater monitor 114 that communicates a current pooltemperature to other devices within the smart home environment 100and/or receives commands for controlling the pool temperature.Similarly, the smart home environment 100 may include an irrigationmonitor 116 that communicates information regarding irrigation systemswithin the smart home environment 100 and/or receives controlinformation for controlling such irrigation systems.

By virtue of network connectivity, one or more of the smart home devicesmay further allow a user to interact with the device even if the user isnot proximate to the device. For example, a user may communicate with adevice using a computer (e.g., a desktop computer, laptop computer, ortablet) or other portable electronic device (e.g., a smartphone) 166. Awebpage or application may be configured to receive communications fromthe user and control the device based on the communications and/or topresent information about the device's operation to the user. Forexample, the user may view a current set point temperature for a deviceand adjust it using a computer. The user may be in the structure duringthis remote communication or outside the structure.

As discussed above, users may control the smart thermostat and othersmart devices in the smart home environment 100 using anetwork-connected computer or portable electronic device 166. In someexamples, some or all of the occupants (e.g., individuals who live inthe home) may register their devices 166 with the smart home environment100. Such registration may be made at a central server to authenticatethe occupant and/or the device as being associated with the home and togive permission to the occupant to use the device to control the smartdevices in the home. Occupants may use their registered devices 166 toremotely control the smart devices of the home, such as when an occupantis at work or on vacation. The occupant may also use a registered deviceto control the smart devices when the occupant is actually locatedinside the home, such as when the occupant is sitting on a couch insidethe home. It should be appreciated that instead of or in addition toregistering the devices 166, the smart home environment 100 may makeinferences about which individuals live in the home and are thereforeoccupants and which devices 166 are associated with those individuals.As such, the smart home environment may “learn” who is an occupant andpermit the devices 166 associated with those individuals to control thesmart devices of the home.

In some implementations, in addition to containing processing andsensing capabilities, the devices 102, 104, 106, 108, 110, 112, 114,116, and/or 118 (“the smart devices”) are capable of data communicationsand information sharing with other smart devices, a central server orcloud-computing system, and/or other devices that are network-connected.The required data communications may be carried out using any of avariety of custom or standard wireless protocols (IEEE 802.15.4, Wi-Fi,ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a,WirelessHART, MiWi, etc.) and/or any of a variety of custom or standardwired protocols (CAT6 Ethernet, HomePlug, etc.), or any other suitablecommunication protocol.

In some implementations, the smart devices serve as wireless or wiredrepeaters. For example, a first one of the smart devices communicateswith a second one of the smart devices via a wireless router. The smartdevices may further communicate with each other via a connection to oneor more networks 162 such as the Internet. Through the one or morenetworks 162, the smart devices may communicate with a smart homeprovider server system 164 (also called a central server system and/or acloud-computing system herein). In some implementations, the smart homeprovider server system 164 may include multiple server systems, eachdedicated to data processing associated with a respective subset of thesmart devices (e.g., a video server system may be dedicated to dataprocessing associated with camera(s) 118). The smart home providerserver system 164 may be associated with a manufacturer, support entity,or service provider associated with the smart device. In someimplementations, a user is able to contact customer support using asmart device itself rather than needing to use other communicationmeans, such as a telephone or Internet-connected computer. In someimplementations, software updates are automatically sent from the smarthome provider server system 164 to smart devices (e.g., when available,when purchased, or at routine intervals).

FIG. 2 is a block diagram illustrating a representative networkarchitecture 200 that includes a smart home network 202 in accordancewith some implementations. In some implementations, one or more smartdevices 204 in the smart home environment 100 (e.g., the devices 102,104, 106, 108, 110, 112, 114, 116, and/or 118) combine to create a meshnetwork in the smart home network 202. In some implementations, the oneor more smart devices 204 in the smart home network 202 operate as asmart home controller. In some implementations, a smart home controllerhas more computing power than other smart devices. In someimplementations, a smart home controller processes inputs (e.g., fromthe smart device(s) 204, the electronic device 166, and/or the smarthome provider server system 164) and sends commands (e.g., to the smartdevice(s) 204 in the smart home network 202) to control operation of thesmart home environment 100. In some implementations, some of the smartdevice(s) 204 in the mesh network are “spokesman” nodes (e.g., node204-1) and others are “low-powered” nodes (e.g., node 204-9). Some ofthe smart device(s) 204 in the smart home environment 100 are batterypowered, while others have a regular and reliable power source, such asby connecting to wiring (e.g., to 120V line voltage wires) behind thewalls 154 of the smart home environment. The smart devices that have aregular and reliable power source are referred to as “spokesman” nodes.These nodes are typically equipped with the capability of using awireless protocol to facilitate bidirectional communication with avariety of other devices in the smart home environment 100, as well aswith the central server or cloud-computing system 164. In someimplementations, one or more “spokesman” nodes operate as a smart homecontroller. On the other hand, the devices that are battery powered arereferred to as “low-power” nodes. These nodes tend to be smaller thanspokesman nodes and typically only communicate using wireless protocolsthat require very little power, such as Zigbee, 6LoWPAN, etc.

In some implementations, some low-power nodes are incapable ofbidirectional communication. These low-power nodes send messages, butthey are unable to “listen”. Thus, other devices in the smart homeenvironment 100, such as the spokesman nodes, cannot send information tothese low-power nodes.

As described, the spokesman nodes and some of the low-powered nodes arecapable of “listening.” Accordingly, users, other devices, and/or thecentral server or cloud-computing system 164 may communicate controlcommands to the low-powered nodes. For example, a user may use theportable electronic device 166 (e.g., a smartphone) to send commandsover the Internet to the central server or cloud-computing system 164,which then relays the commands to one or more spokesman nodes in thesmart home network 202. The spokesman nodes drop down to a low-powerprotocol to communicate the commands to the low-power nodes throughoutthe smart home network 202, as well as to other spokesman nodes that didnot receive the commands directly from the central server orcloud-computing system 164.

In some implementations, a smart nightlight 170 is a low-power node. Inaddition to housing a light source, the smart nightlight 170 houses anoccupancy sensor, such as an ultrasonic or passive IR sensor, and anambient light sensor, such as a photo resistor or a single-pixel sensorthat measures light in the room. In some implementations, the smartnightlight 170 is configured to activate the light source when itsambient light sensor detects that the room is dark and when itsoccupancy sensor detects that someone is in the room. In otherimplementations, the smart nightlight 170 is simply configured toactivate the light source when its ambient light sensor detects that theroom is dark. Further, in some implementations, the smart nightlight 170includes a low-power wireless communication chip (e.g., a ZigBee chip)that regularly sends out messages regarding the occupancy of the roomand the amount of light in the room, including instantaneous messagescoincident with the occupancy sensor detecting the presence of a personin the room. As mentioned above, these messages may be sent wirelessly,using the mesh network, from node to node (i.e., smart device to smartdevice) within the smart home network 202 as well as over the one ormore networks 162 to the central server or cloud-computing system 164.

Other examples of low-power nodes include battery-operated versions ofthe smart hazard detectors 104. These smart hazard detectors 104 areoften located in an area without access to constant and reliable powerand may include any number and type of sensors, such as smoke/fire/heatsensors, carbon monoxide/dioxide sensors, occupancy/motion sensors,ambient light sensors, temperature sensors, humidity sensors, and thelike. Furthermore, the smart hazard detectors 104 may send messages thatcorrespond to each of the respective sensors to the other devices and/orthe central server or cloud-computing system 164, such as by using themesh network as described above.

Examples of spokesman nodes include smart doorbells 106, smartthermostats 102, smart wall switches 108, and smart wall plugs 110.These devices 102, 106, 108, and 110 are often located near andconnected to a reliable power source, and therefore may include morepower-consuming components, such as one or more communication chipscapable of bidirectional communication in a variety of protocols.

In some implementations, the smart home environment 100 includes servicerobots 168 that are configured to carry out, in an autonomous manner,any of a variety of household tasks.

FIG. 3 illustrates a network-level view of an extensible devices andservices platform 300 with which the smart home environment 100 of FIG.1 is integrated, in accordance with some implementations. The extensibledevices and services platform 300 includes remote servers or cloudcomputing system 164. Each of the intelligent, network-connected devices102, 104, 106, 108, 110, 112, 114, 116, and 118 from FIG. 1 (identifiedsimply as “devices” in FIGS. 2-4) may communicate with the remoteservers or cloud computing system 164. For example, a connection to theone or more networks 162 may be established either directly (e.g., using3G/4G connectivity to a wireless carrier), or through a networkinterface 160 (e.g., a router, switch, gateway, hub, or an intelligent,dedicated whole-home control node), or through any combination thereof.

In some implementations, the devices and services platform 300communicates with and collects data from the smart devices of the smarthome environment 100. In addition, in some implementations, the devicesand services platform 300 communicates with and collects data from aplurality of smart home environments across the world. For example, thesmart home provider server system 164 collects home data 302 from thedevices of one or more smart home environments, where the devices mayroutinely transmit home data or may transmit home data in specificinstances (e.g., when a device queries the home data 302). Examplecollected home data 302 includes, without limitation, power consumptiondata, occupancy data, HVAC settings and usage data, carbon monoxidelevels data, carbon dioxide levels data, volatile organic compoundslevels data, sleeping schedule data, cooking schedule data, inside andoutside temperature and humidity data, television viewership data,inside and outside noise level data, pressure data, video data, etc.

In some implementations, the smart home provider server system 164provides one or more services 304 to smart homes. Example services 304include, without limitation, software updates, customer support, sensordata collection/logging, remote access, remote or distributed control,and/or use suggestions (e.g., based on the collected home data 302) toimprove performance, reduce utility cost, increase safety, etc. In someimplementations, data associated with the services 304 is stored at thesmart home provider server system 164, and the smart home providerserver system 164 retrieves and transmits the data at appropriate times(e.g., at regular intervals, upon receiving a request from a user,etc.).

In some implementations, the extensible devices and the servicesplatform 300 includes a processing engine 306, which may be concentratedat a single server or distributed among several different computingentities. In some implementations, the processing engine 306 includesengines configured to receive data from the devices of smart homeenvironments (e.g., via the Internet and/or a network interface), toindex the data, to analyze the data and/or to generate statistics basedon the analysis or as part of the analysis. In some implementations, theanalyzed data is stored as derived home data 308.

Results of the analysis or statistics may thereafter be transmitted backto the device that provided home data used to derive the results, toother devices, to a server providing a webpage to a user of the device,or to other non-smart device entities. In some implementations, usestatistics, use statistics relative to use of other devices, usepatterns, and/or statistics summarizing sensor readings are generated bythe processing engine 306 and transmitted. The results or statistics maybe provided via the one or more networks 162. In this manner, theprocessing engine 306 may be configured and programmed to derive avariety of useful information from the home data 302. A single servermay include one or more processing engines.

The derived home data 308 may be used at different granularities for avariety of useful purposes, ranging from explicit programmed control ofthe devices on a per-home, per-neighborhood, or per-region basis (forexample, demand-response programs for electrical utilities), to thegeneration of inferential abstractions that may assist on a per-homebasis (for example, an inference may be drawn that the homeowner hasleft for vacation and so security detection equipment may be put onheightened sensitivity), to the generation of statistics and associatedinferential abstractions that may be used for government or charitablepurposes. For example, processing engine 306 may generate statisticsabout device usage across a population of devices and send thestatistics to device users, service providers or other entities (e.g.,entities that have requested the statistics and/or entities that haveprovided monetary compensation for the statistics).

In some implementations, to encourage innovation and research and toincrease products and services available to users, the devices andservices platform 300 exposes a range of application programminginterfaces (APIs) 310 to third parties, such as charities 314,governmental entities 316 (e.g., the Food and Drug Administration or theEnvironmental Protection Agency), academic institutions 318 (e.g.,university researchers), businesses 320 (e.g., providing devicewarranties or service to related equipment, targeting advertisementsbased on home data), utility companies 324, and other third parties. TheAPIs 310 are coupled to and permit third-party systems to communicatewith the smart home provider server system 164, including the services304, the processing engine 306, the home data 302, and the derived homedata 308. In some implementations, the APIs 310 allow applicationsexecuted by the third parties to initiate specific data processing tasksthat are executed by the smart home provider server system 164, as wellas to receive dynamic updates to the home data 302 and the derived homedata 308.

For example, third parties may develop programs and/or applications,such as web applications or mobile applications, that integrate with thesmart home provider server system 164 to provide services andinformation to users. Such programs and applications may be, forexample, designed to help users reduce energy consumption, topreemptively service faulty equipment, to prepare for high servicedemands, to track past service performance, etc., and/or to performother beneficial functions or tasks.

FIG. 4 illustrates an abstracted functional view 400 of the extensibledevices and services platform 300 of FIG. 3, with reference to aprocessing engine 306 as well as devices of the smart home environment,in accordance with some implementations. Even though devices situated insmart home environments will have a wide variety of different individualcapabilities and limitations, the devices may be thought of as sharingcommon characteristics in that each device is a data consumer 402 (DC),a data source 404 (DS), a services consumer 406 (SC), and a servicessource 408 (SS). Advantageously, in addition to providing controlinformation used by the devices to achieve their local and immediateobjectives, the extensible devices and services platform 300 may also beconfigured to use the large amount of data that is generated by thesedevices. In addition to enhancing or optimizing the actual operation ofthe devices themselves with respect to their immediate functions, theextensible devices and services platform 300 may be directed to“repurpose” that data in a variety of automated, extensible, flexible,and/or scalable ways to achieve a variety of useful objectives. Theseobjectives may be predefined or adaptively identified based on, e.g.,usage patterns, device efficiency, and/or user input (e.g., requestingspecific functionality).

FIG. 4 shows the processing engine 306 as including a number ofprocessing paradigms 410. In some implementations, the processing engine306 includes a managed services paradigm 410 a that monitors and managesprimary or secondary device functions. The device functions may includeensuring proper operation of a device given user inputs, estimating that(e.g., and responding to an instance in which) an intruder is or isattempting to be in a dwelling, detecting a failure of equipment coupledto the device (e.g., a light bulb having burned out), implementing orotherwise responding to energy demand response events, and/or alerting auser of a current or predicted future event or characteristic. In someimplementations, the processing engine 306 includes anadvertising/communication paradigm 410 b that estimates characteristics(e.g., demographic information), desires and/or products of interest ofa user based on device usage. Services, promotions, products or upgradesmay then be offered or automatically provided to the user. In someimplementations, the processing engine 306 includes a social paradigm410 c that uses information from a social network, provides informationto a social network (for example, based on device usage), and/orprocesses data associated with user and/or device interactions with thesocial network platform. For example, a user's status as reported totrusted contacts on the social network may be updated to indicate whenthe user is home based on light detection, security system inactivationor device usage detectors. As another example, a user may be able toshare device-usage statistics with other users. In yet another example,a user may share HVAC settings that result in low power bills and otherusers may download the HVAC settings to their smart thermostat 102 toreduce their power bills.

In some implementations, the processing engine 306 includes achallenges/rules/compliance/rewards paradigm 410 d that informs a userof challenges, competitions, rules, compliance regulations and/orrewards and/or that uses operation data to determine whether a challengehas been met, a rule or regulation has been complied with and/or areward has been earned. The challenges, rules, and/or regulations mayrelate to efforts to conserve energy, to live safely (e.g., reducingexposure to toxins or carcinogens), to conserve money and/or equipmentlife, to improve health, etc. For example, one challenge may involveparticipants turning down their thermostat by one degree for one week.Those participants that successfully complete the challenge arerewarded, such as with coupons, virtual currency, status, etc. Regardingcompliance, an example involves a rental-property owner making a rulethat no renters are permitted to access certain owner's rooms. Thedevices in the room having occupancy sensors may send updates to theowner when the room is accessed.

In some implementations, the processing engine 306 integrates orotherwise uses extrinsic information 412 from extrinsic sources toimprove the functioning of one or more processing paradigms. Theextrinsic information 412 may be used to interpret data received from adevice, to determine a characteristic of the environment near the device(e.g., outside a structure that the device is enclosed in), to determineservices or products available to the user, to identify a social networkor social-network information, to determine contact information ofentities (e.g., public-service entities such as an emergency-responseteam, the police or a hospital) near the device, to identify statisticalor environmental conditions, trends or other information associated witha home or neighborhood, and so forth.

FIG. 5 illustrates a representative operating environment 500 in which avideo server system 508 provides data processing for monitoring andfacilitating review of motion events in video streams captured by videocameras 118. As shown in FIG. 5, the video server system 508 receivesvideo data from video sources 522 (including cameras 118) located atvarious physical locations (e.g., inside homes, restaurants, stores,streets, parking lots, and/or the smart home environments 100 of FIG.1). Each video source 522 may be bound to one or more reviewer accounts,and the video server system 508 provides video monitoring data for thevideo source 522 to client devices 504 associated with the revieweraccounts. For example, the portable electronic device 166 is an exampleof the client device 504.

In some implementations, the smart home provider server system 164 or acomponent thereof serves as the video server system 508. In someimplementations, the video server system 508 is a dedicated videoprocessing server that provides video processing services to videosources and client devices 504 independent of other services provided bythe video server system 508.

In some implementations, each of the video sources 522 includes one ormore video cameras 118 that capture video and send the captured video tothe video server system 508 substantially in real-time. In someimplementations, each of the video sources 522 includes a controllerdevice (not shown) that serves as an intermediary between the one ormore cameras 118 and the video server system 508. The controller devicereceives the video data from the one or more cameras 118, optionallyperforms some preliminary processing on the video data, and sends thevideo data to the video server system 508 on behalf of the one or morecameras 118 substantially in real-time. In some implementations, eachcamera has its own on-board processing capabilities to perform somepreliminary processing on the captured video data before sending theprocessed video data (along with metadata obtained through thepreliminary processing) to the controller device and/or the video serversystem 508.

As shown in FIG. 5, in accordance with some implementations, each of theclient devices 504 includes a client-side module 502. The client-sidemodule 502 communicates with a server-side module 506 executed on thevideo server system 508 through the one or more networks 162. Theclient-side module 502 provides client-side functionality for the eventmonitoring and review processing and communications with the server-sidemodule 506. The server-side module 506 provides server-sidefunctionality for event monitoring and review processing for any numberof client-side modules 502 each residing on a respective client device504. The server-side module 506 also provides server-side functionalityfor video processing and camera control for any number of the videosources 522, including any number of control devices and the cameras118.

In some implementations, the server-side module 506 includes one or moreprocessors 512, a video storage database 514, an account database 516,an I/O interface to one or more client devices 518, and an I/O interfaceto one or more video sources 520. The I/O interface to one or moreclients 518 facilitates the client-facing input and output processingfor the server-side module 506. The account database 516 stores aplurality of profiles for reviewer accounts registered with the videoprocessing server, where a respective user profile includes accountcredentials for a respective reviewer account, and one or more videosources linked to the respective reviewer account. The I/O interface toone or more video sources 520 facilitates communications with one ormore video sources 522 (e.g., groups of one or more cameras 118 andassociated controller devices). The video storage database 514 storesraw video data received from the video sources 522, as well as varioustypes of metadata, such as motion events, event categories, eventcategory models, event filters, and event masks, for use in dataprocessing for event monitoring and review for each reviewer account.

Examples of a representative client device 504 include a handheldcomputer, a wearable computing device, a personal digital assistant(PDA), a tablet computer, a laptop computer, a desktop computer, acellular telephone, a smart phone, an enhanced general packet radioservice (EGPRS) mobile phone, a media player, a navigation device, agame console, a television, a remote control, a point-of-sale (POS)terminal, a vehicle-mounted computer, an ebook reader, or a combinationof any two or more of these data processing devices or other dataprocessing devices.

Examples of the one or more networks 162 include local area networks(LAN) and wide area networks (WAN) such as the Internet. The one or morenetworks 162 are implemented using any known network protocol, includingvarious wired or wireless protocols, such as Ethernet, Universal SerialBus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for MobileCommunications (GSM), Enhanced Data GSM Environment (EDGE), codedivision multiple access (CDMA), time division multiple access (TDMA),Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or anyother suitable communication protocol.

In some implementations, the video server system 508 is implemented onone or more standalone data processing apparatuses or a distributednetwork of computers. In some implementations, the video server system508 also employs various virtual devices and/or services of third partyservice providers (e.g., third-party cloud service providers) to providethe underlying computing resources and/or infrastructure resources ofthe video server system 508. In some implementations, the video serversystem 508 includes, but is not limited to, a handheld computer, atablet computer, a laptop computer, a desktop computer, or a combinationof any two or more of these data processing devices or other dataprocessing devices.

The server-client environment 500 shown in FIG. 5 includes both aclient-side portion (e.g., the client-side module 502) and a server-sideportion (e.g., the server-side module 506). The division offunctionality between the client and server portions of operatingenvironment 500 can vary in different implementations. Similarly, thedivision of functionality between a video source 522 and the videoserver system 508 can vary in different implementations. For example, insome implementations, the client-side module 502 is a thin-client thatprovides only user-facing input and output processing functions, anddelegates all other data processing functionality to a backend server(e.g., the video server system 508). Similarly, in some implementations,a respective one of the video sources 522 is a simple video capturingdevice that continuously captures and streams video data to the videoserver system 508 with limited or no local preliminary processing on thevideo data. Although many aspects of the present technology aredescribed from the perspective of the video server system 508, thecorresponding actions performed by a client device 504 and/or the videosources 522 would be apparent to one of skill in the art. Similarly,some aspects of the present technology may be described from theperspective of a client device or a video source, and the correspondingactions performed by the video server would be apparent to one of skillin the art. Furthermore, some aspects of the present technology may beperformed by the video server system 508, a client device 504, and avideo sources 522 cooperatively.

FIG. 6 is a block diagram illustrating a video server system 508 inaccordance with some implementations. The video server system 508typically includes one or more processing units (CPUs) 512, one or morenetwork interfaces 604 (e.g., including the I/O interface to one or moreclients 504 and the I/O interface to one or more video sources 522),memory 606, and one or more communication buses 608 for interconnectingthese components (sometimes called a chipset). The memory 606 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM, or otherrandom access solid state memory devices. In some implementations, thememory 606 includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. In some implementations, the memory 606 includes one ormore storage devices remotely located from the one or more processingunits 512. The memory 606, or alternatively the non-volatile memorywithin the memory 606, comprises a non-transitory computer readablestorage medium. In some implementations, the memory 606, or thenon-transitory computer readable storage medium of the memory 606,stores the following programs, modules, and data structures, or a subsetor superset thereof:

-   -   an operating system 610, including procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 612 for connecting the video        server system 508 to other computing devices (e.g., the client        devices 504 and the video sources 522 including camera(s) 118)        connected to the one or more networks 162 via the one or more        network interfaces 604 (wired or wireless);    -   a server-side module 506, which provides server-side data        processing and functionality for the event monitoring and        review, including but not limited to:        -   an account administration module 614 for creating reviewer            accounts, performing camera registration processing to            establish associations between video sources to their            respective reviewer accounts, and providing account            login-services to the client devices 504;        -   a video data receiving module 616 for receiving raw video            data from the video sources 522, and preparing the received            video data for event processing and long-term storage in the            video storage database 514;        -   a camera control module 618 for generating and sending            server-initiated control commands to modify the operation            modes of the video sources, and/or receiving and forwarding            user-initiated control commands to modify the operation            modes of the video sources 522;        -   an event detection module 620 for detecting motion event            candidates in video streams from each of the video sources            522, including motion track identification, false positive            suppression, and event mask generation and caching;        -   an event categorization module 622 for categorizing motion            events detected in received video streams;        -   a zone creation module 624 for generating zones of interest            in accordance with user input;        -   a person identification module 626 for identifying            characteristics associated with the presence of humans in            the received video streams;        -   a filter application module 628 for selecting event filters            (e.g., event categories, zones of interest, a human filter,            etc.) and applying the selected event filters to past and            new motion events detected in the video streams;        -   a zone monitoring module 630 for monitoring motion within            selected zones of interest and generating notifications for            new motion events detected within the selected zones of            interest, where the zone monitoring takes into account            changes in the surrounding context of the zones and is not            confined within the selected zones of interest;        -   a real-time motion event presentation module 632 for            dynamically changing characteristics of event indicators            displayed in user interfaces as new event filters, such as            new event categories or new zones of interest, and for            providing real-time notifications as new motion events are            detected in the video streams; and        -   an event post-processing module 634 for providing summary            time-lapse for past motion events detected in video streams,            and providing event and category editing functions to users            for revising past event categorization results; and    -   server data 636, which includes data for use in data processing        of motion event monitoring and review. In some implementations,        this includes one or more of:        -   a video storage database 514 storing raw video data            associated with each of the video sources 522 (each            including one or more cameras 118) of each reviewer account,            as well as event categorization models (e.g., event            clusters, categorization criteria, etc.), event            categorization results (e.g., recognized event categories,            and assignment of past motion events to the recognized event            categories, representative events for each recognized event            category, etc.), event masks for past motion events, video            segments for each past motion event, preview video (e.g.,            sprites) of past motion events, and other relevant metadata            (e.g., names of event categories, locations of the cameras            118, creation time, duration, DTPZ settings of the cameras            118, etc.) associated with the motion events; and        -   an account database 516 for storing account information for            reviewer accounts, including login-credentials, associated            video sources, relevant user and hardware characteristics            (e.g., service tier, camera model, storage capacity,            processing capabilities, etc.), user interface settings,            monitoring preferences, etc.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 606 stores a subset of the modules and data structures identifiedabove. In some implementations, the memory 606 stores additional modulesand data structures not described above.

FIG. 7 is a block diagram illustrating a representative client device504 associated with a reviewer account in accordance with someimplementations. The client device 504 typically includes one or moreprocessing units (CPUs) 702, one or more network interfaces 704, memory706, and one or more communication buses 708 for interconnecting thesecomponents (sometimes called a chipset). The client device 504 alsoincludes a user interface 710. The user interface 710 includes one ormore output devices 712 that enable presentation of media content,including one or more speakers and/or one or more visual displays. Theuser interface 710 also includes one or more input devices 714,including user interface components that facilitate user input such as akeyboard, a mouse, a voice-command input unit or microphone, a touchscreen display, a touch-sensitive input pad, a gesture capturing camera,or other input buttons or controls. Furthermore, the client device 504optionally uses a microphone and voice recognition or a camera andgesture recognition to supplement or replace the keyboard. In someimplementations, the client device 504 includes one or more cameras,scanners, or photo sensor units for capturing images. In someimplementations, the client device 504 includes a location detectiondevice 715, such as a GPS (global positioning satellite) or othergeo-location receiver, for determining the location of the client device504.

The memory 706 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices. Insome implementations, the memory 706 includes non-volatile memory, suchas one or more magnetic disk storage devices, one or more optical diskstorage devices, one or more flash memory devices, or one or more othernon-volatile solid state storage devices. In some implementations, thememory 706 includes one or more storage devices remotely located fromthe one or more processing units 702. The memory 706, or alternativelythe non-volatile memory within the memory 706, comprises anon-transitory computer readable storage medium. In someimplementations, the memory 706, or the non-transitory computer readablestorage medium of memory 706, stores the following programs, modules,and data structures, or a subset or superset thereof:

-   -   an operating system 716, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 718 for connecting the client        device 504 to other computing devices (e.g., the video server        system 508 and the video sources 522) connected to the one or        more networks 162 via the one or more network interfaces 704        (wired or wireless);    -   a presentation module 720 for enabling presentation of        information (e.g., user interfaces for application(s) 726 or the        client-side module 502, widgets, websites and web pages thereof,        and/or games, audio and/or video content, text, etc.) at the        client device 504 via the one or more output devices 712 (e.g.,        displays, speakers, etc.) associated with the user interface        710;    -   an input processing module 722 for detecting one or more user        inputs or interactions from one of the one or more input devices        714 and interpreting the detected input or interaction;    -   a web browser module 724 for navigating, requesting (e.g., via        HTTP), and displaying websites and web pages thereof, including        a web interface for logging into a reviewer account, controlling        the video sources associated with the reviewer account,        establishing and selecting event filters, and editing and        reviewing motion events detected in the video streams of the        video sources;    -   one or more applications 726 for execution by the client device        504 (e.g., games, social network applications, smart home        applications, and/or other web or non-web based applications);    -   a client-side module 502, which provides client-side data        processing and functionality for monitoring and reviewing motion        events detected in the video streams of one or more video        sources, including but not limited to:        -   an account registration module 728 for establishing a            reviewer account and registering one or more video sources            with the video server system 508;        -   a camera setup module 730 for setting up one or more video            sources within a local area network, and enabling the one or            more video sources to access the video server system 508 on            the Internet through the local area network;        -   a camera control module 732 for generating control commands            for modifying an operating mode of the one or more video            sources in accordance with user input;        -   an event review interface module 734 for providing user            interfaces for reviewing event timelines, editing event            categorization results, selecting event filters, presenting            real-time filtered motion events based on existing and newly            created event filters (e.g., event categories, zones of            interest, a human filter, etc.), presenting real-time            notifications (e.g., pop-ups) for newly detected motion            events, and presenting smart time-lapse of selected motion            events;        -   a zone creation module 736 for providing a user interface            for creating zones of interest for each video stream in            accordance with user input, and sending the definitions of            the zones of interest to the video server system 508; and        -   a notification module 738 for generating real-time            notifications for all or selected motion events on the            client device 504 outside of the event review user            interface; and    -   client data 770 storing data associated with the reviewer        account and the video sources 522, including, but not limited        to:        -   account data 772, which includes information related to the            reviewer account, and the video sources, such as cached            login credentials, camera characteristics, user interface            settings, display preferences, etc.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, modules or datastructures, and thus various subsets of these modules may be combined orotherwise re-arranged in various implementations. In someimplementations, the memory 706 stores a subset of the modules and datastructures identified above. In some implementations, the memory 706stores additional modules and data structures not described above.

In some implementations, at least some of the functions of the videoserver system 508 are performed by the client device 504, and thecorresponding sub-modules of these functions may be located within theclient device 504 rather than the video server system 508. In someimplementations, at least some of the functions of the client device 504are performed by the video server system 508, and the correspondingsub-modules of these functions may be located within the video serversystem 508 rather than the client device 504. The client device 504 andthe video server system 508 shown in FIGS. 6-7, respectively, are merelyillustrative, and different configurations of the modules forimplementing the functions described herein are possible in variousimplementations.

FIG. 8 is a block diagram illustrating a representative camera system118 in accordance with some implementations. Sometimes the camera system118 is referred to herein as a “camera” 118. In some implementations,the camera system 118 includes one or more processing units 802 (e.g.,CPUs, ASICs, FPGAs, or microprocessors), one or more communicationinterfaces 804, memory 806, and one or more communication buses 808 forinterconnecting these components (sometimes called a chipset). In someimplementations, the camera 118 includes one or more input devices 810such as one or more buttons for receiving input and one or moremicrophones. In some implementations, the camera 118 includes one ormore output devices 812 such as one or more indicator lights, a soundcard, a speaker, a small display for displaying textual information anderror codes, etc. In some implementations, the camera 118 includes alocation detection device 814, such as a GPS (global positioningsatellite) or other geo-location receiver, for determining the locationof the camera 118.

As illustrated in FIGS. 10-12 below, the camera includes a sensor array852 that captures video images, and a plurality of illuminators 856,which illuminate a scene when there is insufficient ambient light.Typically, the illuminators emit infrared (IR) light. In someimplementations, the camera 118 includes one or more optional sensors854, such as a proximity sensor, a motion detector, an accelerometer, ora gyroscope.

In some implementations, the camera includes one or more radios 850. Theradios 850 enable radio communication networks in the smart homeenvironment and allow the camera 118 to communicate wirelessly withsmart devices using one or more of the communication interfaces 804. Insome implementations, the radios 850 are capable of data communicationsusing any of a variety of custom or standard wireless protocols (e.g.,IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart,ISA100.11a, WirelessHART, MiWi, etc.), custom or standard wiredprotocols (e.g., Ethernet, HomePlug, etc.), and/or any other suitablecommunication protocol.

The communication interfaces 804 include, for example, hardware capableof data communications (e.g., with home computing devices, networkservers, etc.), using any of a variety of custom or standard wirelessprotocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave,Bluetooth Smart, ISA100.11a, WirelessHART, MiWi, etc.) and/or any of avariety of custom or standard wired protocols (e.g., Ethernet, HomePlug,USB, etc.), or any other suitable communication protocol.

The memory 806 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices. Insome implementations, the memory 806 includes non-volatile memory, suchas one or more magnetic disk storage devices, one or more optical diskstorage devices, one or more flash memory devices, or one or more othernon-volatile solid state storage devices. The memory 806, oralternatively the non-volatile memory within the memory 806, comprises anon-transitory computer readable storage medium. In someimplementations, the memory 806, or the non-transitory computer readablestorage medium of the memory 806, stores the following programs,modules, and data structures, or a subset or superset thereof:

-   -   an operating system 816, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 818, which connects the camera        118 to other computing devices (e.g., the video server system        508, a client device 504, network routing devices, one or more        controller devices, and networked storage devices) connected to        the one or more networks 162 via the one or more communication        interfaces 804 (wired or wireless);    -   a video control module 820, which modifies the operation mode        (e.g., zoom level, resolution, frame rate, recording and        playback volume, lighting adjustment, AE and IR modes, etc.) of        the camera 118, enabling/disabling the audio and/or video        recording functions of the camera 118, changing the pan and tilt        angles of the camera 118, resetting the camera 118, and so on;    -   a video capturing module 824, which captures and generates a        video stream. In some implementations, the video capturing        module sends the video stream to the video server system 508 as        a continuous feed or in short bursts;    -   a video caching module 826, which stores some or all captured        video data locally at one or more local storage devices (e.g.,        memory, flash drives, internal hard disks, portable disks,        etc.);    -   a local video processing module 828, which performs preliminary        processing of the captured video data locally at the camera 118.        For example, in some implementations, the local video processing        module 828 compresses and encrypts the captured video data for        network transmission, performs preliminary motion event        detection, performs preliminary false positive suppression for        motion event detection, and/or performs preliminary motion        vector generation;    -   camera data 830, which in some implementations includes one or        more of:        -   camera settings 832, including network settings, camera            operation settings, camera storage settings, etc.; and        -   video data 834, including video segments and motion vectors            for detected motion event candidates to be sent to the video            server system 508.    -   an illumination module 860, which controls the illuminators 856.        In some implementations, the illumination module 860 identifies        low-light conditions and turns on illuminators as needed. In        some implementations, the illumination module controls the        illuminators 856 individually. Some implementations store one or        more illumination patterns, which are used when the illumination        module is used by the depth mapping module 878;    -   an image capture module 862, which uses the image sensor array        852 to capture images. In some implementations, the image        capture module 852 can capture either IR images 864 or RGB        images 866. Typically, the camera 118 is capable of capturing        both still images as well as video streams;    -   a lookup table generation module 868, which uses captured images        872 to generate lookup tables 874, as illustrated in FIGS. 13,        14, 15A, and 15B. The lookup tables are subsequently used by the        depth mapping module 878 to construct depth maps 876 of a scene.        In some implementations, the lookup table generation module 868        includes a normalization module 880, which is used to normalize        the vectors in the lookup tables;    -   one or more databases 870, which store various data used by the        camera 118. In some implementations, the database stores        captured images 872, including IR images 864 and/or RGB images        866. In some implementations, the image capture module 862        stores captured IR images 864 and RGB images 866 temporarily        (e.g., in volatile memory) before being stored more permanently        in the database 870. In some implementations, the database 870        stores lookup tables 874, which are used by the depth mapping        module to generate depth maps 876. In some implementations, the        computed depth maps 876 are also stored in the database 870; and    -   a depth mapping module 878, which uses the lookup tables 874 to        build one or more depth maps 876 as described below with respect        to FIGS. 16A-16D, 17A, and 17B.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 806 stores a subset of the modules and data structures identifiedabove. In some implementations, the memory 806 stores additional modulesand data structures not described above.

In some implementations, at least some of the functions of the camera118 are performed by a client device 504, the server system 508, and/orone or more smart devices 204, and the corresponding sub-modules ofthese functions may be located within the client device 504, the serversystem 508, and/or smart devices 204, rather than the camera 118.Similarly, in some implementations, at least some of the functions ofthe client device, the server system, and/or smart devices are performedby the camera 118, and the corresponding sub-modules of these functionsmay be located within the camera 118. For example, in someimplementations, a camera 118 captures an IR image of an illuminatedscene (e.g., using the illumination module 860 and the image capturemodule 862), while a server system 508 stores the captured images (e.g.,in the video storage database 514) and creates a depth map 876 based onthe captured images (e.g., performed by a depth mapping module 878stored in the memory 606). The server system 508, the client device 504,and the camera 118, shown in FIGS. 6-8 are merely illustrative, anddifferent configurations of the modules for implementing the functionsdescribed herein are possible in various implementations.

FIG. 9 is a block diagram illustrating a scene understanding server 900.A scene understanding server 900 is commonly part of a video serversystem 508. In some implementations, the functionality of a sceneunderstanding server 900 is included with other functionality providedby a video server system. A scene understanding server 900 may be one ormore physically separate computing devices, or may be incorporated intoa server that provides other functionality as well.

A scene understanding server 900 typically includes one or moreprocessing units (CPUs) 902 for executing modules, programs, orinstructions stored in the memory 914 and thereby performing processingoperations; one or more network or other communications interfaces 904;memory 914; and one or more communication buses 912 for interconnectingthese components. The communication buses 912 may include circuitry(sometimes called a chipset) that interconnects and controlscommunications between system components. In some implementations, theserver 900 includes a user interface 906, which may include a displaydevice 908 and one or more input devices 910, such as a keyboard and amouse.

In some implementations, the memory 914 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices. In some implementations, the memory 914 includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. In some implementations, thememory 914 includes one or more storage devices remotely located fromthe CPU(s) 902. The memory 914, or alternately the non-volatile memorydevice(s) within the memory 914, comprises a non-transitory computerreadable storage medium. In some implementations, the memory 914, or thecomputer readable storage medium of memory 914, stores the followingprograms, modules, and data structures, or a subset thereof:

-   -   an operating system 916, which includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a communications module 918, which is used for connecting the        server 900 to other computers via the one or more communication        network interfaces 904 (wired or wireless) and communication        networks 162, such as the Internet, other wide area networks,        local area networks, metropolitan area networks, and so on;    -   a display module 920, which receives input from one or more        input devices 910, and generates user interface elements for        display on a display device 908;    -   a lookup table generation module 868, as described above in FIG.        8 with respect to a camera 118. In some implementations, the        lookup table generation module includes a normalization module        880;    -   a depth mapping module 878, as described above in FIG. 8 with        respect to a camera 118;    -   one or more object classifiers 922, which classify objects in        the field of vision of a camera 118. Some implementations        include a window detection module 924, which identifies regions        of a scene as probable windows. The window detection module 924        is described below with respect to FIGS. 18A-18E. Some        implementations include a floor/wall/ceiling module 926, which        identifies regions of a scene as floors, walls, and ceilings.        The floor/wall/ceiling module 926 is described below with        respect to FIGS. 20A-20K, 25A, and 25B. In some implementations,        the floor/wall/ceiling module 926 uses a depth map constructed        by the depth mapping module 878 as described below with respect        to FIGS. 16A-16D, 17A, 17B, and 23A-23C. In some        implementations, the floor/wall/ceiling module 926 uses the        depth map to construct an x-direction depth gradient G_(x) 940        and a y-direction gradient G_(y) 942, and uses these to        construct a depth edge map 944. In some implementations, the        floor/wall/ceiling module 926 uses the depth edge map 944 to        identify closed components 946, as illustrated in FIG. 20F        below. For each of these components, some implementations fit a        plane 948, as illustrated below with respect to FIGS. 20H-20J.        If the fitted plane 948 is a good fit and is facing in the        proper direction, it is identified as a probable floor, wall, or        ceiling. This is described below with respect to FIGS. 20A-20K,        25A, and 25B. Some implementations have classifiers in addition        to the window and floor classifiers 924 and 926;    -   some implementations include a zone correction module 928, which        uses depth maps generated at different times to determine if the        camera 118 has moved. If a user has set up zones of interest in        the scene, the zone correction module 928 is able to use the        original zone definition together with the computed camera        movement to determine an adjusted definition of the zone based        on the new camera position. This is described below with respect        to FIGS. 19A-19I. In some implementations, the zone correction        module creates point clouds 930 using the depth maps, and        computes a transformation that maps the first point cloud to the        second point cloud;    -   some implementations include a camera pose estimator 932, which        estimates the position of the camera 118 with respect to the        room in which it is located. In some implementations, the camera        position includes the estimated height of the camera 118 (i.e.,        the height of the image sensor array) as well as the angle of        altitude. In some implementations, an angle of zero represents a        camera that is pointed exactly horizontal (e.g., parallel to the        floor), with positive angles when the camera is pointing down        and negative angles when the camera is pointing up. One of skill        in the art recognizes that alternative coordinate systems can be        used as well, such as a reference angle of 0 representing a        camera 118 pointing directly down and a reference angle of 180        pointing directly up. The operation of the camera pose estimator        is described below with respect to FIGS. 21A-21E; and    -   one or more databases 870, which store captured images 872,        lookup tables 874, and/or depth maps 876, as described above in        FIG. 8 with respect to a camera 118. In some implementations,        the captured images 872 include both RGB images 934 and IR        images 936.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various implementations. In some implementations, thememory 914 stores a subset of the modules and data structures identifiedabove. In some implementations, the memory 914 stores additional modulesand data structures not described above.

In some implementations, at least some of the functions of the sceneunderstanding server 900 are performed by a client device 504, thecamera 118, or other servers in the video server system 508. Similarly,in some implementations, at least some of the functions of the clientdevice 504, the video server system 508, and the camera 118 areperformed by the scene understanding server 900. For example, in someimplementations, a camera 118 captures an IR image of an illuminatedscene (e.g., using the illumination module 860 and the image capturemodule 862), while a scene understanding server 900 stores the capturedimages 872 and creates one or more depth maps 876 based on the capturedimages (e.g., performed by a depth mapping module 878).

FIG. 10 provides a front view of a camera 118, in accordance with someimplementations. The camera 118 includes a sensor array 852, a pluralityof illuminators 856 (e.g., the illuminators 856-1 to 856-8), and anenclosure 1010. In this particular implementation, the array 852 ofimage sensors (which are typically photodiodes) is centrally located andrectangular, but this configuration is not required. An actual imagesensor array 852 typically has a much higher resolution than shown inthe illustration. In this implementation, there are eight illuminatorsthat are grouped into four pairs, with one pair for each of: top,bottom, left, and right. In other implementations, there are more offewer illuminators, and the illuminators may be grouped in differentways (or not grouped at all). In some implementations, the camera 118includes camera circuitry and/or other camera components that are notillustrated in this figure.

As described in greater detail below, the illuminators 856 are activatedto illuminate a scene by emitting streams of light (e.g., infrared (IR)light). During illumination, light rays are scattered by and reflect offof object surfaces in the scene (e.g., walls, furniture, humans, etc.).Reflected light rays are then detected by the sensor array 852, whichcaptures an image of the scene (e.g., and IR image or an RGB image). Thecaptured image digitally measures the intensity of the reflected IRlight for each of the pixels in the sensor array 852.

In some implementations, the illuminators 856 are light emitting diodes(LEDs). In some implementations, the illuminators 856 are semiconductorlasers or other semiconductor light sources. In some implementations,the illuminators 856 are configured to emit light spanning a broad rangeof the electromagnetic spectrum, including light in the IR range (e.g.,700 nm to 1 mm), the visible light range (e.g., 400 nm-700 nm), and/orthe ultraviolet range (e.g., 10 nm-400 nm). In some implementations, aportion of the illuminators 856 are configured to emit light in a firstrange (e.g., IR range), while other illuminators 856 are configured toemit light in a second range (e.g., visible light range). In someimplementations, the illuminators 856 are configured to emit light inaccordance with one or more predefined illumination patterns. Forexample, in some implementations, the illumination pattern is circularround-robin in a clockwise order. In some of these implementations, theround-robin pattern activates two illuminators at a time, as illustratedin FIG. 14 below. An illumination pattern may specify other parametersas well, such as the length of time each illuminator is activated, theoutput power (e.g., measured in watts), or other parameters.

The sensor array 852 converts an optical image (e.g., reflected lightrays) into an electric signal. In some implementations, the sensor array852 is a CCD image sensor, a CMOS sensor, or another type of lightsensor device (e.g., a hybrid of CCD and CMOS). The sensor array 852includes a plurality of individual light-sensitive sensors. In someimplementations, the sensors of the sensor array 852 are arranged in arectangular grid pattern as illustrated in FIG. 10. Upon exposure tolight, each sensor of the sensor array 852 detects a measurable andproportional value corresponding to the light intensity. In someimplementations, the sensor array 852 or other camera circuitry convertsthe measured value (e.g., current) into a digital value. In someimplementations, the sensor array 852 or the enclosure 1010 includes anIR filter to remove wavelengths of incident light that fall outside of apredefined range. For example, some implementations use an IR filterthat passes only light having wavelengths in the range of 810 nm to 870nm. In some implementations, the illuminators 856 emit light at aspecified wavelength and the light reaching the sensor array is filteredto correspond to the specified wavelength of the illuminators.

In some implementations, the camera 118 includes additional cameracomponents, such as one or more lenses, image processors, shutters,and/or other components known to those skilled in the art of digitalphotography.

In some implementations, the camera 118 also includes camera circuitryfor coordinating various image capture functionality of the camera 118.In some implementations, the camera circuitry is coupled to theilluminators 856, to the sensor array 852, and/or to other cameracomponents, and coordinates the operational timing of the various cameradevice components. In some implementations, when capturing an IR imageof a scene, the camera circuitry activates a subset of the illuminators856, activates the sensor array 852 to capture the image, and determinesan appropriate shutter speed to manage the image exposure. In someimplementations, the camera circuitry performs basic image processing ofraw images captured by the sensor array 852 during the exposure. Theimage processing includes filtering and conversion of a produced voltageor current at the sensor array 852 into a digital value.

FIG. 11 illustrates just the image sensor array 852 for a camera 118. Inthis example, the sensors 1110 in the sensor array 852 are in arectangular grid of rows and columns. In the illustration, therectangular grid is a square, but other implementations have grids ofsensors 1110 that are not square (e.g., more sensors horizontally thanvertically). Also, the sensors 1110 themselves are not necessarilysquare. In this example, the first row consists of a line of sensors1110 _(1,1), 1110 _(1,2), . . . . The sensor in the ith row and jthcolumn is labeled 1110 _(i,j).

FIG. 12 provides a side view of a camera 118, in accordance with someimplementations. The same components of the camera 118 in FIG. 10 areillustrated in FIG. 12: the illuminators 856, the sensor array 852, andthe enclosure 1010.

In some implementations, one or more illuminators 856 are angledrelative to the planar axis of the sensor array, such as illuminator856-1 in FIG. 12. By positioning the illuminators 856 at respectiveangles (e.g., angle 1210), portions of a scene will be illuminated atgreater or lesser intensities depending on which of the illuminators 856are activated and the angles at which the activated illuminators arepositioned. FIGS. 16A-16D illustrate a sequence of IR images withdifferent illuminators activated. For example, FIG. 16A is an imagecaptured with the top two illuminators activated, whereas FIG. 16C is animage captured with the bottom two illuminators activated.

FIGS. 13-15B illustrate a method for generating a lookup table, which islater used to construct a depth map of a scene in accordance with someimplementations. In some implementations, a lookup table is constructedfor each pixel in the sensor array based on simulating a surface andcomputing an expected intensity of reflected light based on thesimulated surface and a pre-selected illumination pattern. In someimplementations, the physical sensors of the sensor array 852 aregrouped together to simulate an array with a smaller number of pixels.For example, some implementations downsample a 1 megapixel array toabout 10,000 pixels by grouping each 10×10 subarray of sensors into asingle downsampled pixel. In this example, 100 physical sensors of thearray are treated as a single pixel for purposes of building the lookuptable and subsequently using the lookup table. In the followingdescription, the term “pixel” will be used to describe the basic unitfor a table lookup (each pixel corresponds to a lookup table) regardlessof whether the pixel corresponds to a single physical sensor in thesensor array or multiple physical sensors in the sensor array.

To generate a lookup table for a pixel, the lookup table generationmodule 868 determines an expected reflected light intensity at the pixelbased on the simulated surfaces 1304 being at various fixed distances1302 from the pixel. This is illustrated in FIG. 13, with fixeddistances d₁ 1302-1, d₂ 1302-2, d₃ 1302-3, . . . , d_(m) 1302-m, andsurfaces 1304-1, 1304-2, 1304-3, . . . , 1304-m. The number of distinctsimulated distances 1302 affects the accuracy of the subsequentlyestimated depths. In this example, all of the surfaces 1304 are planar.In other implementations, the surfaces are spherical, parabolic, cubic,or other appropriate shape. Typically, however, all of the surfaces areof the same type (e.g., there would generally not be a mixture of planarand spherical surfaces). In the simulation, each virtual surface has aconstant surface reflectivity.

For each depth 1302, the illuminators 856 of the camera 118 aresimulated to activate in accordance with a pre-defined illuminationpattern. An illumination pattern specifies the grouping of illuminators856 (if any), specifies the order the groups of illuminators areactivated, and may specify other parameters related to the operation ofthe illuminators. FIG. 14 provides an example in which the illuminators856 are grouped into consecutive pairs in a clockwise orientation andactivated in that order. At a first time 1402-1, the top illuminationgroup 1404-1 is activated, at a second time 1402-2 a second illuminationgroup 1404-2 is activated, at a third time 1402-3 a third illuminationgroup 1404-3 is activated, and at a fourth time 1402-4 a fourthillumination group 1404-4 is activated. In the example illustrated inFIG. 14, there are four illumination groups 1404 in the illuminationpattern, so there are four distinct estimated light intensity values.

In some implementations, the estimated light intensity values are placedinto an intensity matrix Y_(i,j) 1506, as illustrated in FIG. 15A. Inthis matrix, each column corresponds to one depth, and each rowcorresponds to an illumination group from the illumination pattern. Forexample, the first column 1500-1 corresponds to a first distance d₁. Thefirst light intensity estimate 1501-1 corresponds to the firstillumination group 1404-1, the second light intensity estimate 1502-1corresponds to the second illumination group 1404-2, the third lightintensity estimate 1503-1 corresponds to the third illumination group1404-3, and the fourth light intensity estimate 1504-1 corresponds tothe fourth illumination group 1404-4.

The kth column 1500-k in the intensity matrix Y_(i,j) 1506 has fourlight intensity estimates 1501-k, 1502-k, 1503-k, and 1504-k,corresponding to the same four illumination groups in the illuminationpattern. Finally, the mth column 1500-m has four list intensityestimates corresponding to the same four illumination groups in theillumination pattern. Note that the matrix Y_(i,j) 1506 is for a singlepixel i,j (e.g., as downsampled from the sensor array 852).

As currently computed, the entries in the intensity matrix Y_(i,j) 1506depend on the reflectivity ρ of the simulated surface. Because differentactual surfaces have varying reflectivities, it would be useful to“normalize” the matrix in a way that eliminates the reflectivityconstant ρ. In some implementations, the columns of the intensity matrixY_(i,j) 1506 are normalized by dividing the elements of each column bythe length (e.g., L₂ norm) of the column.

FIG. 15B illustrates normalizing the kth column Y_(i,j) (k) 1508 of thematrix 1506. The normalized column {tilde over (Y)}_(i,j)(k) 1510 iscomputed from the column Y_(i,j)(k) 1508 by dividing each component bythe length ∥Y_(i,j)(k)∥₂=√{square root over (y_(1k) ²+y_(2k) ²+y₃₅²+y_(4k) ²)}. Performing the same normalization process for each columnin the intensity matrix Y_(i,j) 1506 creates a normalized lookup table{tilde over (Y)}_(i,j).

Note that after normalization, each column of the lookup table {tildeover (Y)}_(i,j) has the same normalized length, even though each columncorresponds to a different distance from the sensor array. However, thedistribution of values across the elements (corresponding to theillumination groups) are different for different depths (e.g., thenormalized first column is different from the normalized kth column).

Some implementations take advantage of symmetry to reduce the number oflookup tables. For example, using the illumination pattern illustratedin FIG. 14, some implementations reduce the number of lookup tables by afactor of four (e.g., using rotational symmetry), or reduce the numberof lookup tables by a factor of eight (e.g., using rotational symmetryand reflection symmetry).

FIGS. 16A-16D, 17A, and 17B illustrate a method for creating a depthmap, in accordance with some implementations. The depth map estimatesthe depth of objects in a scene. The scene is typically all or part ofthe field of vision of a camera 118. The depth map is created for a2-dimensional array of pixels. In some implementations, the pixelscorrespond to the individual image sensors in the image sensor array852. In some implementations, each pixel corresponds to plurality ofindividual image sensors from the array 852. For example, in someimplementations, each pixel corresponds to a 10×10 square subarray ofimage sensors.

FIGS. 16A-16D illustrate a sequence of captured images 1606 of anilluminated scene. In each of these figures, the scene is illuminated bya different group of illuminators 856 of the camera 118 in accordancewith an illumination pattern. Typically, the illumination pattern usedfor generating the lookup tables is used again for creating a depth mapof a scene. That is, the illuminators are grouped into the sameillumination groups, are activated in the same order, and use the sameparameters (e.g., power and length of activation). As shown in FIGS.16A-16D, each illumination group focuses light on a different portion ofthe scene. For example, the illumination group 1602-1 in FIG. 16Acreates a brighter portion 1604-1 at the top of the scene, and theillumination group 1602-3 in FIG. 16C creates a brighter portion 1604-3at the bottom of the scene. Similarly, the illumination group 1602-2 inFIG. 16C creates a brighter portion 1604-2 on the right side of thescene and the illumination group 1602-4 in FIG. 16D creates a brighterportion on the left side of the scene. In the example of FIGS. 16A-16D,there are four captured images 1606-1, 1606-2, 1606-3, and 1606-4 basedon the four illumination groups 1602. In addition, a fifth image iscaptured when none of the illuminators are activated.

As illustrated in FIG. 17A, a vector {right arrow over (b)}_(i,j) 1706is constructed for each pixel i,j. The four components of the vector1706 correspond to the four distinct illumination groups 1602-1, 1602-2,1602-3, and 1602-4. The first component b₁−b₀ 1701 is the lightintensity b₁ at the pixel when the first illumination group 1602-1 isactive minus the light intensity b₀ at the pixel from the baselineimage. Similarly, the second component b₂−b₀ 1702 is the light intensityb₂ at the pixel when the second illumination group 1602-2 is activeminus the light intensity b₀ at the pixel from the baseline image. Thethird component b₃−b₀ 1703 is the light intensity b₃ at the pixel whenthe third illumination group 1602-3 is active minus the baseline lightintensity b₀, and the fourth component b₄−b₀ 1704 is the light intensityb₄ at the pixel when the fourth illumination group 1602-4 is activeminus the baseline light intensity b₀.

For each individual pixel there is a separate lookup table, which isgenerated as described above by simulating virtual surfaces at differentdepths. The actual depth in the scene at the pixel is determined byfinding the closest matching record in the lookup table for the pixel.In this example, the vector {right arrow over (b)}_(i,j) 1706 and therecords in the lookup table (e.g., column {tilde over (Y)}_(i,j) (k)1510) are four dimensional vectors. In some implementations, the closestmatch is computed by finding the lookup table record whose “direction”in R⁴ most closely aligns with the sample vector {right arrow over(b)}_(i,j) 1706. This can be determined by computing the inner product(e.g., dot product) of the vector {right arrow over (b)}_(i,j) 1706 witheach of the records in the lookup table. In some implementations, theinner product of the vector {right arrow over (b)}_(i,j) 1706 with therecord {tilde over (Y)}_(i,j)(k) 1510 is

{right arrow over (b)}_(i,j), {tilde over (Y)}_(i,j)(k)

=y_(1k)(b₁−b₀)+y_(2k)(b₂−b₀)+y_(3k)(b₃−b₀)+y_(4k)(b₄−b₀). The record inthe lookup table whose inner product with the sample vector 1706 is thegreatest has an associated depth (i.e., the simulated depth for whichthe lookup table record was created), and this is the estimated depthfor the pixel. Typically, the inner product used is just the dotproduct, as illustrated in this example.

The process just described is shown concisely by the formula in FIG.17B. The lookup table index {circumflex over (k)} is estimated bycomputing the dot product of the normalized lookup table records {tildeover (Y)}_(i,j)(k) 1510 with the sample vector {right arrow over(b)}_(i,j) 1706, and selecting the index for which the dot product ismaximal. The estimated depth corresponds to the index {circumflex over(k)}.

In the example illustrated in FIGS. 16A-16D, 17A, and 17B, the eightilluminators are grouped into four illumination groups. However, manyother illumination patterns are possible with the same set of eightilluminators. For example, in some implementations, the eightilluminators are activated individually, creating lookup tables witheight rows and vectors with eight components. Some implementations useother illumination patterns as well. For example, some implementationsuse two illuminators at a time, but use each illuminator in two groups(e.g., a first group consisting of illuminators 1 and 2, a second groupconsisting of illuminators 2 and 3, a third group consisting ofilluminators 3 and 4, etc.).

FIGS. 18A-18E illustrate a process of identifying windows in a scenethat is monitored by a camera 118. FIG. 18A is an RGB image of a scene(illustrated here in black and white) as viewed by a surveillance camera118. Although a human can easily recognize the windows from the RGBphoto, it is more difficult for a computing device to identify thewindows automatically.

In some implementations, the camera 118 has infrared illuminators 856,which illuminate the scene (typically at night) and capture one of moreIR images to form an IR intensity image 1802, as illustrated in FIG.18B. In this example of an IR intensity image 1802, black representshigh intensity and white represents low intensity. Because windows arespecular, the light emitted from the IR illuminators 856 mostly reflectsoff in other directions rather than back towards the image sensor array852 of the camera 118, thus creating regions of low intensity. As seenin the IR intensity image 1802, there are various areas 1804, 1806,1808, and 1810 of low intensity. The low intensity pixels are clusteredtogether to form contiguous regions. In addition to being specular,windows typically have a reasonable size (e.g., a house would not have awindow that is one inch wide), and are generally rectangular. Because ofthe deformation of the images, a rectangular window appears as aquadrilateral, which may not be a rectangle.

Using size and/or quadrilateral analysis of the low intensity regions inFIG. 18B, the process determines that the lower regions 1808 and 1810 donot appear to be windows. However, the upper left low intensity region1804 is sufficiently large and fits in a quadrilateral 1812 fairly well,as indicated in FIG. 18C. Therefore, the region 1804 is designated as aprobable window. Similarly, the upper right low intensity region 1806 issufficiently large and fits well into a quadrilateral 1814, so it isidentified as a probable window as well.

The same techniques described with respect to windows can identify othertypes of objects as well. For example, the same analysis used forwindows can be applied to identify mirrors or television screens. Insome implementations, a sufficiently large quadrilateral region with lowintensity of reflected IR light is identified as a television ratherthan a window based on other information, such as frequent movementwithin the region. Certain materials have reflectivities that areintermediate between a specular surface and a surface with highlydiffused reflections. In some implementations, these materials areidentified by a range of expected image intensity from reflecting the IRlight.

In some implementations, quadrilateral fitting measures the absolutedifference between the quadrilateral and the region, and determines thatthere is a good fit when the absolute difference is less than athreshold percentage of the area of the quadrilateral (e.g., less than5%, less than 10%, or less than 20%). In some implementations, theprocess uses more general polygons rather than quadrilaterals.

Some implementations use motion discontinuity as a factor in determiningwhether a low intensity region is a window. For example, motion of anobject on an opposite side of a window will show up as discontinuousboth as the object enters the field of the window and when the objectexits the field of the window. In some implementations, the presence ofmotion discontinuity within a region is used as evidence that the regionis a window, but the absence of motion discontinuity is not used asevidence that the region is not a window.

FIGS. 18D and 18E are IR images that illustrate two specular regionsthat are probable windows. The dark regions 1822 and 1824 in FIG. 18Dshow up as dark because the IR light from the illuminators is reflectedin a specular way by the windows. In this example, low intensity regionsappear dark, which is the opposite of the display presented in FIG. 18B.Other surfaces create diffused reflection, in which the incoming lightis reflected in all directions, including back to the light source. Thedark regions 1822 and 1824 in FIG. 18D are overlaid by quadrilaterals1832 and 1834 in FIG. 18E. Even though there is some curvatureintroduced by the wide angle lens of the camera, the quadrilaterals 1832and 1834 fit the dark regions 1822 and 1824 fairly well, so they areidentified as probable windows.

FIG. 19A provides an outline for computing zone correction according tosome implementations. Initially, a user defines (1980) a zone ofinterest in a scene while the camera 118 is in a first position 1988. Insome implementations, the zone is defined using a captured RGB image. Insome implementations, the zone is defined using a captured IR image. Insome implementations, zones must be polygons, but other implementationsallow for broader zone definition. Zones of interest are commonly usedfor motion alerts. In some implementations, a camera 118 is notpermanently affixed to a structure, so the camera 118 may move(intentionally or unintentionally). When the camera moves, thepreviously defined zone is no longer valid. Therefore, someimplementations include a zone correction module 928 to compute anadjusted zone that corresponds to the zone originally defined by theuser.

Some implementations build (1982) a depth map based on IR imagescaptured while the camera 118 is in the first position 1988. In someimplementations, the IR images are captured temporally proximate to thetime the zone is defined in order to ensure that the depth map is builtbased on the same field of vision. In some implementations, temporalproximity is defined to be within 12 hours or within 24 hours. At somepoint later, the camera moves (1984). For example, a person may bump thecamera or a person may choose to move the camera slightly to get bettercoverage of a room. Later, some implementations build (1986) a seconddepth map based on IR images captured while the camera 118 is in asecond position 1990. Note that the zone correction module 928 does notnecessarily know the camera has moved. In some implementations, depthmaps are created on a periodic basis (e.g., once each night, every twodays, or once each week).

In some implementations, the zone correction module 928 computes pointclouds 930 corresponding to each of the depth maps, where each point ina point cloud 930 is a three dimensional position in the scene monitoredby the camera, as illustrated below in FIGS. 19B-19I. In someimplementations, a predetermined number of points are selected for eachpoint cloud (e.g., 50, 100, or 1000 points), but in otherimplementations, the number of points varies based on the objects in themonitored scene. In some implementations, the points for each pointcloud are selected based on designated positions within the image sensorarray (e.g., the intersection of each tenth row with each tenth column).In some implementations, the points in the point cloud are selected bydownsampling from the depth map (i.e., combine multiple points from thedepth map to create an individual point for the point cloud). In someimplementations, points for each point cloud are selected based on othercharacteristics, such as proximity to the camera (e.g., choose pointsfrom the depth map that are close to the camera).

The process of comparing two point clouds is sometimes referred to as“registration” by those of skill in the art. A registration processdetermines how to transform one point cloud into another point cloud.Some implementations use one or more iterated closest point (ICP)methods to determine the transformation. When one of the point cloudscan be transformed to match the other point cloud, the iterative processbuilds the transformation as a sequence of steps that converge on thefinal transformation. When the two point clouds are fundamentallydifferent (e.g., from IR images captured from different scenes), theiterative process is generally unable to converge.

After the transformation is determined, the transformation is applied tothe zone defined by the user, thereby creating an adjusted zone thatcorresponds to the defined zone. This is illustrated below in FIGS. 19Gand 19H. In some implementations, the user is prompted to confirm theadjusted zone. The process of performing zone correction is alsodescribed below with respect to the flowchart 2600 in FIGS. 26A-26C.

FIGS. 19B and 19C provide an example of identifying movement of a camera118. In FIG. 19B, certain points are identified in the scene 1900-B. Insome implementations, the points are identified using a depth map of thescene, as described in FIGS. 23A-23C below (e.g., selecting certainpoints that are closer to the camera 118 than nearby points). In someimplementations, at least some of the points are selected based on adepth transition and/or color transition (using an RGB imagecorresponding to the depth map).

In the scene 1900-B of FIG. 19B, seven points have been identified: thepoints 1901 and 1902 that appear to be the left side corners of apicture frame or window; the point 1903 at the left side of an apparenttable; the point 1904 that appears to be the bottom of a table leg, andthree points 1905, 1906, and 1907 that are at various locations on whatappears to be a chair. Note that what the points represent is notrelevant to the analysis. Here, the relative positions of the points(horizontally, vertically, and depth from the camera) identify points in3 dimensional space. In FIG. 19C, seven similar points 1911-1917 havebeen identified, and in this case the depths (not shown) areapproximately the same as the corresponding points 1901-1907 in FIG.19B. However, the scene 1900-B appears to have shifted to the left tocreate the modified scene 1900-C. Rather than concluding that the wholescene has shifted to the left, the zone correction module 928 determinesthat the camera has moved a little to the right. In someimplementations, the points 1901-1907 and the points 1911-1917 arestored as point clouds 930. Although the example of FIGS. 19B and 19Chas a one-to-one correspondence between the points in the two pointclouds 930, the zone correction module 928 does not require such aperfect correspondence between the two point clouds 930.

FIGS. 19D and 19E illustrate detecting camera movement of a differentsort. FIG. 19D is the same as FIG. 19B, with the same seven points1901-1907, but also identifies the distance 1908 between the two points1901 and 1902. The seven points 1921-1927 in FIG. 19E correspond to theseven points 1901-1907 in FIG. 19D, but the depths are now different andthe orientations are a little distorted. For example, the distance 1928between the points 1921 and 1922 in FIG. 19E appears larger than thedistance 1908 in FIG. 19D. Based on the cloud of points 1901-1907 inFIG. 19D and the cloud of points 1921-1927 in FIG. 19E, it appears thatthe scene 1900-D has rotated toward the left (counterclockwise if viewedfrom above) to create the scene 1900-E in FIG. 19E. The zone correctionmodule 928 determines that the camera has been rotated a little to theright to create the different scene perspective.

As illustrated in FIGS. 19B-19E, the zone correction module uses twopoint clouds 930 that represent the field of vision of the camera, anddetermines whether the two point clouds correspond to slightly differentviews of the same scene. In some instances, the camera is moved to acompletely different scene (e.g., a different room), so the two depthmaps are quite different. The zone correction module 930 is generallyable to determine that the point clouds 930 do not correspond.

FIG. 19F illustrates a top view perspective of a camera movement and howcorrelating two point clouds is used to identify the movement. In thisillustration, a camera is initially at a first location 1940, and thenis moved a little to a second location 1950. Because FIG. 19F shows atop view perspective, differences in height above the floor are notdepicted. However, the techniques described here (and in FIGS. 26A-26Cbelow) identify movement of the camera in any direction and/or rotation.

When the camera is at the first location 1940, the field of vision ofthe camera is illustrated by the dotted lines 1942 on the left and 1944on the right. When the camera is at the second location 1950, the fieldof vision of the camera is illustrated by the dotted lines 1952 on theleft and 1954 on the right. A first depth map is created based on imagescaptured while the camera 118 is at the first position 1940, and asecond depth map is created based on images captured while the camera118 is at the second position 1950. For each of the depth maps, a pointcloud is created that contains a plurality of points.

In this illustration, the points 1946-1 and 1946-2 are in the field ofvision of the camera at the first position 1940 but not in the field ofvision from the second location 1950. Conversely, the points 1956-1,1956-2, and 1956-3 are in the field of vision of the camera 118 at thesecond position 1950 but not in the field of vision from the firstlocation 1940. The other points in this illustration are in the sharedregion 1960.

A first point in this region is identified both as point 1946-3 and aspoint 1956-4. The two labels for the same point are due to the presenceof the point in both the first and second depth maps. With respect tothe camera 118, the three dimensional coordinates of the point 1946-3are different from the 3-dimensional coordinates of the point 1956-4,even though the point has not moved. For example, the depth andhorizontal position of the point 1946-3 (as measured from the firstcamera location 1940) are different from the depth and horizontalposition of the point 1956-4 (as measured from the second cameralocation 1950). If the height of the camera above the floor at the firstand second locations are the same, then the measured height of the point1946-3 is the same as the height of the point 1956-4. The same analysisapplies to the second labeled point in the region 1960, which is labeledas both 1946-4 and 1956-5. They are the same physical point in thescene, but have different 3-dimensional coordinates based on the twoviews. The same analysis applies to the third labeled point in theregion 1960, which is labeled as both 1946-5 (from the first depth map)and 1956-6 (from the second depth map).

The first point cloud (containing the points 1946-1-1946-5) iscorrelated to the second point cloud (containing the points1956-1-1956-6), based on points in the overlap region 1960. In practice,the points are not literally identical as they are in this example. Asindicated above, an iterative algorithm determines how to map one of thepoint clouds to the other.

FIG. 19G shows an IR image of a monitored scene, with a zone 1960identified by a user. This zone 1960 outlines an entryway to the roomfrom outside, and thus the user has designated it for motion alerts. Ata later time, a second IR image is captured as illustrated in FIG. 19H.In addition to the IR images illustrated in FIGS. 19G and 19H, thecamera 118 captures sequences of IR images with different sets of IRilluminators activated contemporaneous with the IR images in FIGS. 19Gand 19H. For example, in some implementations, when depth mapping imagesare captured, a first IR image is captured with no illuminatorsactivated, a plurality of additional IR images are captured with varioussubsets of illuminators activated, and a final IR image is captured withall of the illuminators activated. (Of course the image capture is notnecessarily in this order.) As described below with respect to FIGS.23A-23C, the depth mapping module 878 uses the multiple IR images tobuild two depth maps. Points are then selected from each of the depthmaps to form point clouds, and then the two point clouds are registered(aligned) as described above with respect to FIG. 19A and describedbelow with respect to FIGS. 26A-26C.

As shown in FIG. 19H, the uncorrected zone 1962 (using the samecoordinates that were saved for the original zone 1960) no longer coversthe entryway that was covered by the zone 1960 previously. However,using the point clouds created from the depth maps, the zone correctionmodule 928 determines the transformation required to correlate the twoviews, and applies the transformation to the first zone. Thetransformation constructs an adjusted zone 1964, which again covers theentryway. Even if the adjusted zone 1964 is not perfect (it should be alittle wider to match the entryway), it is a much better zone for thecamera in the new position than the uncorrected zone 1962.

FIG. 19I provides a summary of the zone-correction process according tosome implementations. The input 1970 includes a user defined zone, adepth map from an original camera position, and a depth map from a latercamera position. The user-defined zone may be created with respect to anRGB image or an IR image.

When the camera has moved slightly, the process computes an output 1972,which is an adjusted zone. The adjusted zone corresponds to the originalzone, but accounts for the camera movement. This is illustrated aboveFIGS. 19G and 19H.

In some implementations, computing the adjusted zone includes: (1)converting (1974) the original depth map to a point cloud with 3Dcoordinates. In some implementations, the constructed point cloud has atleast 100 points. In some implementations, the point cloud has fewer ormore points. For example, in some implementations, the point cloud has50 points or 500 points. In some implementations, the points for thepoint cloud are randomly or pseudo-randomly selected from the depth map.In some implementations, the points in the point cloud are selected in aregular pattern, such as every tenth pixel horizontally and vertically.In some implementations, the points in the point cloud are selectedbased on specific characteristics, such as proximity to the camera orlocations where there is significant depth discontinuity (see FIGS.20B-20E).

The process builds (1976) a second point cloud from the second map,which corresponds to the current location of the camera. The points inthe second point cloud are generally selected in the same way as for thefirst point cloud.

The process then compares (1978) the two point clouds. This process issometimes referred to as point cloud registration. Some implementationsuse an iterative process to perform point cloud registration. In someimplementations, the process uses an iterated closest point (“ICP”)method. The registration process determines a transformation that mapsthe first point cloud to the second point cloud.

Finally, the process applies (1980) the identified transformation to theuser-selected zone to identify an adjusted zone based on the new cameralocation. In some implementations, the new zone is used immediately. Insome implementations, the user is prompted to confirm the adjusted zone,and the user may tweak the adjusted zone further.

FIGS. 20A-20K illustrate a process performed by a floor/wall/ceilingmodule 946 to identify probable floors, walls, and ceilings. FIG. 20A isan IR image of a scene. Some implementations use a coordinate system inwhich x is measured horizontally, y is measured vertically, and zrepresents the depth into the image from the camera. As illustrated inFIG. 20G, the depth is measured from the camera.

In some implementations, the floor/wall/ceiling module 926 uses a depthmap 876 of the scene, which is constructed as illustrated in FIGS.16A-16D, 17A, 17B, and 23A-23C. The floor/wall/ceiling module 926 usesthe depth map 876 to identify depth discontinuities. In someimplementations, the floor/wall/ceiling module 926 identifies thediscontinuities using an x-direction gradient map G_(x) 940 asillustrated in FIG. 20B and a y-direction gradient map G_(y) 942 asillustrated in FIG. 20C. As illustrated in FIG. 20D, someimplementations combine the two gradients G_(x) 940 and G_(y) 942 toform a binary depth edge map 944, as shown in FIG. 20E. In someimplementations, an edge is identified at a pixel when the total depthchange exceeds a predefined threshold value.

Once the depth discontinuities are identified in the binary depth edgemap 944, the floor/wall/ceiling module 926 identifies the closedcomponents 946 in the image (i.e., regions that are enclosed by theedges). These closed components 946 represent the candidates for floors,walls, and ceilings. FIG. 20F shows the closed components 946corresponding to the depth map 944 in FIG. 20E. The two largestcomponents 946-1 and 946-2 are good candidates. In some implementations,closed components 946 that are smaller than a threshold size areexcluded from further analysis. For example, in some implementations,only the two largest closed components 946-1 and 946-2 are evaluated.

FIG. 20G illustrates how “depth” is measured from the point of view ofthe camera 118. This is a side view of the scene, showing how the depthz correlates to the height y. For example, incident rays 2020-1 to2020-4 have depth that increases as a function of height. This is whatwould be expected for a floor. The incident rays 2020-5 to 2020-8 have adepth that decreases as a function of height. This is what would beexpected for a ceiling.

For each of the closed components 946 that is evaluated, thefloor/wall/ceiling module 926 fits a plane to the points in thecomponent. In some implementations, the fitted plane has an equation ofthe form w_(x)x+w_(y)y+w_(z)z=1, where w_(x), w_(y), and w_(z) areconstants to be determined, as illustrated in FIG. 20H. For each closedcomponent, a subset of points in the component are used to form a matrixC, as illustrated in FIG. 20I. The matrix C has a row for each selectedpoint in the component, and has three columns corresponding to the x, y,and z-coordinates of the points. A single closed component 946 may havea large number of points, so implementations typically take a sampling(e.g., a pseudo-random sample of 20 points or 50 points). The fittedplane 948 should closely match the data, so a “best fit” can bedetermined by measuring the total error. Some implementations use leastsquares, and thus select the values for w_(x), w_(y), and w_(z) tominimize the expression Σ_(i)(w_(x)c_(i1)+w_(y)c_(i2)+w_(z)c_(i3)−1)²,as illustrated in FIG. 20J. Some implementations use alternative methodsto identify a “best” plane for a set of data points from a closedcomponent.

Once a best plane 948 is identified for a component, thefloor/wall/ceiling module 926 evaluates the plane in two ways. First, isthe total error sufficiently small so that the plane is a good fit?Second, does the orientation of the plane correspond to floor, wall, orceiling? Some implementations specify an error threshold, and designatea closed component as a probable floor, wall, or ceiling only when theactual error is less than the threshold. In some implementations, thetotal error is normalized based on the number of points in the sample.

As illustrated in FIG. 20G, a floor should have z increasing as afunction of y. Using the formula in FIG. 20H,

${z = {{{- \frac{w_{y}}{w_{z}}}y} + \left( {{other}\mspace{14mu} {terms}} \right)}},$

so the

${expression} - \frac{w_{y}}{w_{z}}$

should be positive for a floor. Similarly, for a ceiling, the

${expression} - \frac{w_{y}}{w_{z}}$

should be negative. Some implementations also evaluate the magnitude ofthe

${expression} - \frac{w_{y}}{w_{z}}$

to determine whether it is consistent with data expected for a floor orceiling. For walls, the expressions are similar, but use the x-dimensionrather than the y-dimension.

FIG. 20K illustrates that the closed component 946-2 has been identifiedas a probable floor region 2022. In some implementations (notillustrated here), the first closed component 946-1 is identified as awall.

FIGS. 21A-21E illustrate a process for estimating the height andorientation of a video monitoring camera. Typically, the height ismeasured from a floor to the sensor array 852 of the camera 118. Theorientation is measured as an angle with respect to the plane of thefloor. In some implementations, an angle of 0 represents a horizontalorientation and positive angles represent tilting toward the floor (sothat 90 degrees would be pointing straight down). In someimplementations, the “height” is measured relative to a ceiling ratherthan a floor. The techniques described herein with respect to a floorcan be applied in the same way to a ceiling, typically considering thedistance below the ceiling as positive.

In FIG. 21A, the camera 118 is at a height h₁ 2112 above the floor 2110,and some of the floor 2110 is in the field of vision of the camera 118,as illustrated by the dashed lines 2120. In FIG. 21A, the camera isfacing straight forward, so the camera orientation 2116 matches theplane 2118 parallel to the floor 2110. This produces a tilt angle θ₁2114 of 0 degrees.

FIG. 21B shows the same camera 118 at a different height and orientationwith respect to the floor 2110. A portion of the floor 2110 is in thefield of vision of the camera 118, as indicated by the dashed lines2130. The camera 118 is at a height h₂ 2122 above the floor 2110, andthe camera is tilted at an angle θ₂ 2124 of 20 degrees. The angle θ₂ ismeasured between the plane 2128 parallel to the floor and the cameraorientation 2126.

The illustrations in FIGS. 21A and 21B illustrate both the process ofbuilding a dictionary (typically using simulation with varying heightsand tilt angles) as well as determining the position of an actual camera118.

FIG. 21C illustrates a dictionary 2150 of training entries 2152, whichwill be used subsequently to estimate the height and tilt angle of anactual camera 118. In some implementations, the entries 2152 areconstructed by simulating a camera 118 with various heights and tiltangles with respect to a simulated floor. In other implementations, theentries are constructed based on test data with an actual camera 118 atvarious heights and angles relative to an actual floor. In someimplementations, test data is collected in an environment with little orno ambient light so that the collected images are based on just the IRlight emitted by the IR illuminators of the camera.

The dictionary includes a height 2154 and a tilt 2156 for each entry,and includes data for one or more images captured based on differentsets of IR illuminators emitting light. In some implementations, asingle image is captured while all of the IR emitters are on. In someimplementations, a separate image is captured for each individual IRemitter, taken while that IR emitter is on and the remaining IR emittersare off. In some implementations, the emitters are grouped into pairs,as illustrated above with respect to FIG. 14. In the example dictionary2150 in FIG. 21C, there are four subsets (as in FIG. 14), and separateimages 2140, 2142, 2144, and 2146 are simulated or captured for each ofthe subsets. When built using simulation, the estimated intensity ateach pixel depends on the location and orientation of the IR emittersrelative to the image sensor array 852.

In this example dictionary 2150, the second dictionary entry 2152-2corresponds to a height of 0.6 meters and a tilt angle of 10°. In someimplementations, positive title angles indicate the camera is pointingdownward. For this second entry 2152-2, the process simulates orcaptures four images I_(2,1), I_(2,2), I_(2,3), and I_(2,4),corresponding to each of the four subsets of IR illuminators. In someimplementations, abbreviated images are stored. For example, someimplementations store only pixels corresponding to the simulated floor.Note that the pixels in the images are typically downsampled from theimage sensor array. For example, the image sensor array may include 4million individual image sensors, whereas the saved images may includeonly 10,000 pixels.

In this example dictionary 2150, there are 250 dictionary entries 2152,corresponding to heights ranging from 0.6 meters to 3.0 meters (in 0.1meter increments) and angles ranging from 0 degrees to 90 degrees (in 10degree increments). In some implementations, there are fewer or moredictionary entries 2152, depending on the desired granularity, availablestorage space, required processing speed, and/or other considerations.

Whereas a dictionary 2150 is typically creating one time for a givencamera model, the dictionary 2150 can be used many times to estimate theheights and tilt angles of many cameras at many different times.

FIG. 21D illustrates a process for determining the height and tilt angleof an actual camera 118 according to some implementations. When thedictionary 2150 was created, certain distinct subsets of the IRilluminators were specified. The same subsets are used during theestimation process in FIG. 21D. For each of those distinct subsets ofilluminators, the process captures (2160) an IR image (measuring IRlight intensity) while the illuminators in the subset are emitting lightand the IR illuminators not in the subset are not emitting light. Inaddition, the process captures (2160) a baseline light intensity imagewhen none of the IR illuminators are emitting light. The process thencomputes (2162) adjusted IR intensity images for each of the distinctsubsets of IR illuminators by subtracting the baseline intensity imagefrom each of the other images (subtracting on a pixel-by-pixel basis).

Using the adjusted intensity images, the process identifies (2164) atleast one possible floor region. In some implementations, identifying apossible floor region uses techniques illustrated in FIGS. 20A-20K and25A-25B. If no floor regions are identified, some implementationsautomatically switch to determining the position of the camera relativeto the ceiling. When more than one floor region is identified, someimplementations estimate a camera position relative to each of theidentified regions, then select a best fit or compute an aggregatedestimate. If there are two or more regions and the estimates aresimilar, some implementations compute an average or weighted average. Ifthere are two or more regions and the estimates differ substantially,some implementations select the data for the larger height based on thestatistical reasoning that the higher number is more likely to becorrect (e.g., because the smaller number is from a table).

Some implementations use an iterative algorithm for identifying a floorregion. In some of these implementations, the entire set of pixels isused as a starting point for the first iteration, and in each iterationsome of the pixels are removed. In some implementations, the pixelsidentified for removal in each iteration are selected based on overallcontribution to the computed distances between the adjusted IR intensityimages and entries in the dictionary. In some implementations, theprocess combines floor selection (2164) and classification (2166) intoan iterative loop.

Once a floor region is identified, a classifier estimates (2166) the(height, tilt) 2168 using the adjusted IR intensity images, thepreviously computed dictionary 2150, and limiting the analysis to pixelsin the identified floor region. The operation of the classifier isdescribed in more detail in FIG. 21E.

The classifier identifies a “closest” dictionary entry 2152 to theadjusted IR intensity images, and estimates the height and tilt of thecamera based on that closest dictionary entry. When the number ofdictionary entries is small (e.g., 100), some implementations comparethe adjusted IR intensity images to each of the dictionary entries tofind the closest one. In some implementations, the process is able toprune some of the dictionary entries, thereby comparing the adjusted IRintensity images to a smaller list of dictionary entries.

To identify a closest dictionary entry 2152, some implementationscompute distances between vectors, as illustrated in FIG. 21E. In thisfigure, the input is the set of four images I₁, I₂, I₃, and I₄ based onthe different subsets of illuminators, and the baseline image I₀. Thebaseline image I₀ is subtracted from the others to create the input2170, which can be viewed as a long feature vector 2178. In thisexample, each image has n pixels, and the elements are arranged in orderof the images. For example, the elements a₁₁, . . . , a_(1r), . . . ,a_(1n) correspond to the pixels of the image I₁−I₀. In this example, theindex r corresponds to one specific pixel in the identified floorregion. Because there are four distinct images, there are four featurevector components a_(1r), a_(2r), a_(3r), a_(4r) 2174 corresponding tothe rth pixel.

FIG. 21E illustrates comparing the feature vector to the second entry2152-2 in the dictionary 2150. This second entry 2152-2 includesintensity images (I_(2,1), I_(2,2), I_(2,3), I_(2,4)) 2172-2, which canbe represented as a long dictionary entry vector 2180, with componentscorresponding to the components of the feature vector 2178.

To compute the distance between the feature vector 2178 and a dictionaryentry vector 2180, some implementations use Euclidean distance based onthe relevant vector components. The relevant components are the onesassociated with the pixels in the identified floor region. For example,in this case, the rth pixel is part of the identified floor region, sothe four components corresponding to r are included in the calculationof the distance, as illustrated in formula 2176-2. If there are fourilluminator subsets and 100 pixels in the identified floor region, thenthe distance calculation will use 400 components of the vectors. In someimplementations, alternative distance metrics are used, such as thetotal absolute difference between vector components |a_(1r)−b_(1r)|+ . .. or the maximum absolute difference between vector components.

In some implementations, the single closest dictionary entry is used toestimate the camera position. For example, if the second dictionaryentry 2152-2 above is determined to be closer than all of the otherdictionary entries, then the camera is estimated to be at a height of0.6 meters and at an angle of 10 degrees (see FIG. 21C). In someimplementations, the k closest dictionary entries are identified for apredefined positive integer k. These k entries are then used to estimatethe height and tilt angle for the camera. For example, someimplementations compute a weighted average from the k nearest entries,and weight each entry inversely based on its calculated distance. Someimplementations use alternative techniques, such as other regressionalgorithms.

FIGS. 22A-22C provide a flowchart of a process 2200, performed by acomputing device, for generating (2202) a lookup table for use inestimating spatial depth in a visual scene. The method is performed(2204) at a computing device (e.g., a scene understanding server 900)having one or more processors and memory. The memory stores (2204) oneor more programs configured for execution by the one or more processors.

The process identifies (2206) a plurality of distinct subsets of IRilluminators 856 of a camera system 118. One example is illustratedabove in FIGS. 16A-16D, where the camera's 8 illuminators 856 aregrouped into four distinct subsets. One of skill in the art recognizesthat many other alternatives are possible, such as having oneilluminator in each subset, having some overlap between subsets, orhaving different subsets with different numbers of illuminators.

The camera also has (2208) a 2-dimensional array 852 of image sensors.The 2 dimensional array 852 is typically laid out in a rectangularpattern, as illustrated above in FIGS. 10 and 11, but the disclosedprocess 2200 can be applied regardless of the pattern to lay out theimage sensors in the array. In some implementations, the array of imagesensors includes (2210) more than 1,000,000 individual image sensors(e.g., 2²⁴ sensors). The IR illuminators 856 are (2212) in fixedlocations relative to the array 852 of image sensors, as illustrated inFIGS. 10 and 12 above.

The process partitions (2214) the image sensors into a plurality ofpixels. In some implementations, each pixel includes (2216) a respectivesingle image sensor. In some implementations, each pixel includes (2218)a respective plurality of image sensors. In some implementations, eachpixel includes (2220) more than 50 respective image sensors. These are afew ways that implementations partition the individual image sensorsinto pixels. Typically the array of image sensors has a high resolution,but sensors are downsampled to create a more manageable number of pixels(e.g., 10,000 pixels).

A separate lookup table is constructed for each pixel. Each record in alookup table corresponds to a depth in front of the pixel. The accuracyof subsequent depth estimation depends on the number of depths used tobuild each lookup table. For example, if depth data is created for eachinch in front of the pixel, then subsequent depth estimation may beaccurate within an inch. However, if there are only two depth datapoints, the accuracy for subsequent estimation will be limited.

For each pixel, and for each of m distinct depths from the pixel, theprocess performs (2222) the following operations. The process simulates(2224) a virtual surface at the respective depth. Implementations usevarious shapes for the virtual surfaces, such as planar (2226),spherical (2228), parabolic (2230), or cubic (2232). FIG. 13 illustratesthe case of planar surfaces. Typically, an implementation uses the samesurface shape for each of the pixels and depths, although potentiallywith different parameters. For example, when spherical surfaces areused, some implementations simulate a sphere whose radius is the givendepth so that the surfaces at each of the depths create concentricspheres.

For each pixel and for each of the depths (2222), the process alsodetermines (2234) an expected IR light intensity at the respective pixelbased on the respective depth, the shape of the virtual surface, andwhich subset of IR illuminators is emitting IR light. In someimplementations, the expected IR light intensity at the respective pixelis (2236) based on other characteristics of the IR illuminators of thecamera system as well. For example, in some implementations, thecharacteristics include (2238) the lux of the IR illuminators 856. Insome implementations, the characteristics include (2240) orientation ofthe IR illuminators relative to the sensor array. This is illustratedabove in FIG. 12, with illuminator 856-1 oriented at an angle 1210. Insome implementations, the characteristics include (2242) location of theIR illuminators relative to the sensor array.

For each pixel and for each of the depths (2222), the process also forms(2244) an intensity vector using the expected IR light intensity foreach of the distinct subsets. This is illustrated in FIG. 17A above.Typically a baseline value is subtracted from each of the values, wherethe baseline value is measured when none of the illuminators areemitting light. The process then normalizes (2246) the intensity vector.In some implementations, the process normalizes each intensity vector bydetermining (2248) a respective magnitude of the intensity vector anddividing each component of the intensity vector by the respectivemagnitude.

The process constructs (2250) a lookup table for each pixel using thenormalized vectors corresponding to the pixel. Each lookup tableassociates (2252) each respective normalized vector in the table withthe respective depth of the respective simulated surface. Someimplementations use this lookup table as described below with respect tothe process 2300 illustrated in FIGS. 23A-23C.

FIGS. 23A-23C provide a flowchart of a process 2300, performed by acomputing device, for creating (2302) a depth map of a scene. The methodis performed (2304) at a computing device (e.g., a scene understandingserver 900) having one or more processors and memory. The memory stores(2304) one or more programs configured for execution by the one or moreprocessors. In some implementations, the computing device is (2306) aserver distinct from a camera system. In other implementations, thecomputing device is (2308) included in the camera system.

In some implementations, the process 2300 detects (2310) a triggerevent. In some implementations, creating the depth map of the firstscene is (2310) in response to detecting the trigger event. In someimplementations, the first scene includes (2312) a first objectpositioned at a first location within the first scene and the process2300 detects (2314) the first object positioned at a second locationwithin the first scene, where the second location is distinct from thefirst location. The movement of the first object triggers the buildingof the depth map. In some implementations, the trigger event is (2316) apower outage (e.g., build or rebuild the depth map when the computingdevice reboots).

In some implementations, the process 2300 switches (2318) the mode ofoperation of the camera system when building the depth map. For example,some implementations switch (2318) from a first mode of the camerasystem to a second mode of the camera system, including deactivating thefirst mode and activating the second mode. In some implementations, thearray of image sensors has (2320) an associated first pixel gain curvewhen the first mode is activated, and the array of image sensors has(2320) an associated second pixel gain curve when the second mode isactivated.

For each of a plurality of distinct subsets of IR illuminators of thecamera system, the process 2300 performs (2322) a set of operations. Insome implementations, one or more of the subsets of the IR illuminatorsconsists (2324) of a single IR illuminator. In some implementations, theplurality of IR illuminators are orientated (2326) at a plurality ofdistinct angles relative to the array of image sensors. In someimplementations, each of the distinct subsets of IR illuminatorscomprises (2328) two adjacent IR illuminators, and the distinct subsetsof IR illuminators are (2328) non-overlapping. One of skill in the artrecognizes that various groupings, arrangements, and/or configurationsmay be used for the IR illuminators.

The process 2300 receives (2330) a captured IR image of a first scenetaken by a 2-dimensional array of image sensors of the camera systemwhile the respective subset of IR illuminators are emitting IR light andthe IR illuminators not in the respective subset are not emitting IRlight. This occurs for each distinct subset of IR illuminators. Theimage sensors are partitioned (2332) into a plurality of pixels. Asnoted above with respect to the process 2200 in FIG. 22A, thepartitioning of image sensors into pixels can occur in various waysdepending on the implementation. In some implementations, the process2300 receives (2334) a baseline IR image of the scene captured by thearray of sensors while none of the IR illuminators are emitting IRlight. Some implementations subtract the light intensity from thisbaseline image from the light intensity in each of the other captured IRimages, as illustrated above in FIG. 17A.

For each of the pixels, the process 2300 performs (2336) severaloperations, including using (2338) the captured IR images to form arespective vector of light intensity at the respective pixel. In someimplementations, the respective vector for each pixel has (2340) aplurality of components. Each of the components corresponds (2340) to arespective IR light intensity for the respective pixel for a respectivecaptured IR image. This is illustrated above in FIG. 17A, where thevector {right arrow over (b)}_(i,j) 1706 has four components,corresponding to the four illumination groups 1602 illustrated in FIGS.16A-16D. In some implementations, forming each respective vector oflight intensity at a respective pixel comprises (2342) subtracting alight intensity at the pixel in the baseline IR image from the lightintensity at the pixel in each of the captured IR images, as illustratedin FIG. 17A. In this way, the vector measures the additional light thatis received at the image sensor array 852 based on reflections of lightemitted from each of the illumination groups.

For each pixel (2336), the process 2300 then estimates (2344) a depth inthe first scene at the respective pixel by looking up the respectivevector in a respective lookup table. In some implementations, theprocess looks up (2346) the respective vector in the respective lookuptable by computing (2346) an inner product of the respective vector withrecords in the lookup table. One of skill in the art recognizes that ina vector space an inner product can be used to measure the extent towhich a pair of vectors are pointing in the same direction. In someinstances, the inner product is (2350) an ordinary dot product. In someimplementations, the process 2300 computes (2348) the inner product ofthe respective vector with each respective record in the respectivelookup table. In some implementations, fewer than all of the innerproducts are computed for the lookup table (e.g., based on optimizationtechniques, such as recognizing that certain records in the lookup tablewould produce smaller inner products than some inner products that arealready computed).

In some implementations, the process 2300 determines (2352) the depth inthe first scene at the pixel as the depth corresponding to a record inthe lookup table whose inner product with the respective vector isgreatest among the computed inner products for the respective vector.This is illustrated above with respect to FIG. 17B.

In some implementations, the respective lookup table is generated (2354)during a calibration process at the camera 118. In some implementations,the calibration process includes (2356) simulating a virtual planarsurface at a plurality of respective depths in the first scene. In someimplementations, the calibration process includes (2358), for each pixeland each respective depth, determining an expected reflected lightintensity. In some implementations, each respective lookup table isdownloaded (2362) to the camera system 118 from a remote server duringan initialization process prior to creating the depth map.

In some implementations, each respective lookup table includes (2360) aplurality of normalized light intensity vectors, where each normalizedlight intensity vector corresponds to a respective depth in the firstscene. This is illustrated above in FIGS. 13, 14, 15A, and 15B.

Although lookup tables have been identified separately for each pixel,one of skill in the art recognizes that the separate logical lookuptables are not necessarily stored as separate files or databases. Forexample, some implementations store all of the lookup tables as a singlephysical table in a relational database or as a single physical file ona file server. In some implementations, the totality of lookup tables isstored as a small number of distinct files. As described above,implementations generate and use the lookup tables on various devicesdepending on the capabilities of the camera system 118, availablenetwork bandwidth, and other resources. For example, for camera systemswith limited processing power and/or storage, some implementations buildand use the lookup tables at a scene understanding server 900. Thecamera system 118 captures the IR images (e.g., baseline image plusadditional images with different sets of illuminators on), and transmitsthem to the server 900. The server then constructs the depth map. Insome implementations, the lookup tables are constructed at the server900 based on the depth simulations and knowledge of the cameraconfiguration, and then downloaded to the camera. In some of theseimplementations, the camera 118 uses the lookup tables itself to build adepth map.

FIGS. 24A-24C provide a flowchart of a process 2400, performed by acomputing device, for classifying (2402) objects in a scene. The methodis performed (2404) at a computing device (e.g., a scene understandingserver 900) having one or more processors and memory. The memory stores(2404) one or more programs configured for execution by the one or moreprocessors. In some implementations, the computing device is (2406) aserver distinct from a camera system. In other implementations, thecomputing device is (2408) included in the camera system.

The process receives (2410) a captured IR image of a scene taken by a2-dimensional image sensor array of a camera system while one or more IRilluminators of the camera system are emitting IR light, thereby formingan IR intensity map of the scene with a respective intensity valuedetermined for each pixel of the IR image. Typically, the IR image iscaptured at night, so most of the intensity is based on reflection ofthe light from the IR illuminators. Typical surfaces disperse light inall directions, so some of the emitted light is reflected back to theimage sensor array. For a specular surface, however, such as a window,mirror, or some television screens, the incoming light at a surface isreflected off primarily in one direction, with the angle of incidenceequal to the angle of reflection. A specular region therefore typicallyhas low intensity in the IR intensity map.

The pixels in the IR intensity map can correspond to the image sensorsin the array 852 in various ways, as previously illustrated with respectto FIG. 22A (boxes 2214-2220). In some implementations, each pixel ofthe IR image corresponds (2412) to a unique respective image sensor inthe image sensor array. In some implementations, the pixels of the IRimage form (2414) a partition of the image sensors in the image sensorarray. In some of these implementations, at least one pixel corresponds(2416) to a plurality of image sensors in the image sensor array.

Typically, the camera system 118 includes (2418) a plurality of IRilluminators, as illustrated above in FIGS. 10 and 12. In someimplementations, the process 2400 constructs the IR intensity map frommultiple distinct IR images. For example, in some implementations, theprocess receives (2420) a respective IR sub-image of the scene for eachof a plurality of distinct subsets of IR illuminators of the camerasystem. Each sub-image is captured (2420) while illuminators in arespective subset are emitting IR light and the IR illuminators not inthe respective subset are not emitting IR light. The process 2400computes (2422) an average of the intensity values at the pixel in eachof the sub-images to determine the intensity value for the pixel.

The process uses (2424) the IR intensity map to identify a plurality ofpixels whose corresponding intensity values are within a predefinedintensity range. In some implementations, the predefined intensity rangeis (2426) all intensity values below a threshold value. This is theintensity range typically used when the goal is to identify windows.Some implementations use other ranges to identify other specificmaterials.

The process 2400 clusters (2428) the identified plurality of pixels(i.e., the pixels identified based on the intensity range) into one ormore regions that are substantially contiguous. This is illustratedabove with respect to FIG. 18B. Some implementations use other factorsin the clustering process as well. For example, some implementations seta threshold size for a region. Small regions of low intensity are eithercombined with other nearby regions or ignored. In some implementations,clustering the identified plurality of pixels into one or more regionsuses (2430) a depth map that was constructed using the image sensorarray. For example, when trying to identify windows, a window should becontinuous. A single region with two or more significantly disparatedepths is not likely to be a window. In some implementations, clusteringthe identified plurality of pixels into one or more regions uses (2432)an RGB image of the scene captured using the image sensor array. Forexample, evaluating the color distribution of a region can identify someregions that are unlikely to be windows (e.g., the presence of certaincolors or the number of distinct colors).

The process 2400 determines (2434) that a first region of the one ormore regions corresponds to a specific material based, at least in part,on the intensity values of the pixels in the first region. In someimplementations, determining that a first region of the one or moreregions corresponds to a specific material includes (2436) determiningthat the first region is substantially a quadrilateral. This isillustrated by the quadrilaterals 1812 and 1814 in FIG. 18C above, andthe quadrilaterals 1832 and 1834 in FIG. 18E. In some implementations,the first region is (2438) substantially a quadrilateral when a totalabsolute difference in area between the first region and thequadrilateral is less than a threshold percentage of the quadrilateral'sarea (e.g., less that 10% of the area of the quadrilateral). In someimplementations, the specific material is (2440) glass and the firstregion is determined to correspond to a window in the scene. In someimplementations, the region is identified as a probable windowcandidate, which is subsequently confirmed either by a user or otherindependent criteria.

Once a region has been classified, the process 2400 stores (2442)information in the memory that identifies the region. The informationcan be stored in various ways. In some implementations, the process 2400stores coordinates for the region, such as coordinates of a centroid, orcoordinates of a subset of points along the boundary. In someimplementations, the process 2400 creates a two-dimensional scene mapcorresponding to the pixels, and specifies a value (e.g., a number or acharacter) to identify the object/material/function for each pixel. Forexample, in some implementations, a value of 0 indicates no information,a value of 1 indicates a probable window, 2 indicates a probable floor,3 indicates a probable wall, and 4 indicates a probable ceiling. Usageof a scene map is illustrated in FIG. 30 below. Identification offloors, walls, and ceilings is described above with respect to FIGS.20A-20K and below with respect to FIGS. 25A-25B. Some implementationsuse characters instead of numbers, such as a “W” to indicate a probablewindow, an “F” to indicate a probable floor, and a blank space if thereis no information about a possible object at the pixel.

In some implementations, the process 2400 receives (2444) a video streamof the scene from the camera system and reviews (2446) the video streamto detect movement in the scene. Movement in the scene can be used toidentify possible intruders in a home or other potential problems. Insome implementations, the first region is excluded (2446) from movementdetection. For example, if the first region is identified as a window,movement in the window region may be movement on the other side of thewindow (e.g., outside), and thus not suitable for a motion alert. Inanother example, the first region is a television set, and thus “motion”in the region is typically based on displayed television images ratherthan real motion at the scene. In some implementations, the process 2400generates (2448) a motion alert when there is motion detected at thescene outside of the first region.

FIGS. 25A-25B provide a flowchart of a process 2500, performed by acomputing device, for identifying (2502) large planar objects in scenes.The method is performed (2504) at a computing device (e.g., a sceneunderstanding server 900) having one or more processors and memory. Thememory stores (2504) one or more programs configured for execution bythe one or more processors. In some implementations, the computingdevice is (2506) a server distinct from a camera system. In otherimplementations, the computing device is (2508) included in the camerasystem.

The process 2500 receives (2510) a plurality of captured IR images of ascene taken by a 2-dimensional array of image sensors of a camerasystem. Each IR image is captured (2512) when illuminators in a distinctsubset of IR illuminators of the camera system 118 are emitting light.In some implementations, the image sensors are partitioned (2514) into aplurality of pixels. As described above with respect to FIG. 22A (e.g.,boxes 2214-2220), implementations group the image sensors into pixels invarious ways.

The process 2500 constructs (2516) a depth map of a scene using theplurality of IR images. Some implementations use a process as describedin FIGS. 23A-23C (process 2300) to construct the depth map. In someimplementations, for each pixel the process 2500 performs (2518) a setof operations. In some implementations, the set of operations includesusing (2520) the captured IR images to form a respective vector of lightintensity at the respective pixel. In some implementations, the set ofoperations includes estimating (2522) a depth in the first scene at therespective pixel using the respective vector and a respective lookuptable. In some implementations, lookup tables are constructed using aprocess as described in FIGS. 22A-22C (process 2200).

The process 2500 uses (2524) the depth map to compute a binary depthedge map 944 for the scene. The binary depth edge map 944 identifies(2524) which points in the depth map comprise depth discontinuities.This is illustrated in FIGS. 20B-20D above. The process 2500 thenidentifies (2526) a plurality of contiguous components based on thebinary depth edge map. This is illustrated in FIG. 20E above. Depthdiscontinuities create boundaries between components.

The process then determines (2528) that a first component of theplurality of contiguous components represents a large planar surface inthe scene. This determination involves a few steps. A first step is tofit (2530) a plane to the points in the first component. In someimplementations, the fitting uses least squares to find the best planefor the data in the component. Some implementations use other techniquesto identify a “best” plane for the data, such as minimizing the sum ofabsolute differences between a hypothetical plan and the points in thecomponent. Implementations typically use a sampling of data points froma component to fit the best plane. For example, some implementations use50 or 100 sample data points from a component.

In making the determination that the first component represents a largeplanar surface, the process also confirms that the “best” plane isactually a good plane for the data. In some implementations, the process2500 determines (2540) that the plane fitting residual error is lessthan a predefined threshold. In some implementations, the plane fittingresidual error is the sum of the absolute differences between the planeand the sample points in the component. In some implementations, theplane fitting residual error is the sum of the squares of thedifferences between the sample points and the plane, or the square rootof the sum of the squares. In some implementations, the plane fittingresidual error is the maximum absolute difference between the samplepoints and the plane. Some implementations use two or more techniques toconfirm that the residual error is small (e.g., the maximum absoluteerror is less than a first threshold and the sum of the absolute errorsis less than a second threshold).

Once the plane is fitted and it is determined that the residual error issufficiently small, the first component is identified as a large planarsurface. The process 2500 then analyzes the plane to determine whetherthe surface is likely to be a floor, a ceiling, or a wall. To make thisdetermination, some implementations determine (2532) the orientation ofthe plane. This is illustrated above with respect to FIG. 20G. When theorientation of the plane is upwards, the process 2500 determines (2534)that the plane is probably a floor. When the orientation of the plane isdownwards, the process 2500 determines (2536) that the plane is probablya ceiling. When the orientation of the plane is horizontal, the process2500 determines (2538) that the plane is probably a wall.

Some implementations use other criteria as well in making thedetermination that a component represents a large planar surface. Forexample, some implementations require the component to have a minimumthreshold area to be classified as a probable floor, wall, or ceiling.

FIGS. 26A-26C provide a flowchart of a process 2600, performed by acomputing device, for recomputing (2602) zones in scenes based onphysical movement of a camera. The method is performed (2604) at acomputing device (e.g., a scene understanding server 900) having one ormore processors and memory. The memory stores (2604) one or moreprograms configured for execution by the one or more processors. In someimplementations, the computing device is (2606) a server distinct from acamera system. In other implementations, the computing device is (2608)included in the camera system.

The process 2600 receives (2610) a first RGB image of a scene taken by a2-dimensional array of image sensors of a camera system at a first time.The RGB image identifies what is in the field of vision of the camera.The process also receives (2612) a first plurality of distinct IR imagesof the scene taken by the array of image sensors temporally proximate tothe first time. In general, the temporal proximity ensures that thefield of vision of the camera while capturing the IR images issubstantially the same as the field of vision of the camera whilecapturing the RGB image. Commonly, the RGB image is captured duringdaylight hours, whereas the IR images are captured at night. In someimplementations, temporal proximity means within 24 hours or 12 hours.Each of the IR images is taken (2614) while a different subset of IRilluminators of the camera system is emitting light.

The process 2600 uses (2616) the first plurality of IR images toconstruct a first depth map of the scene, where the first depth mapindicates a respective depth in the scene at a plurality of pixels. Someimplementations use a process like the depth mapping process 2300described with respect to FIGS. 23A-23C to construct the first depthmap. The pixels of the depth map correspond to the image sensors of thearray. In some implementations, each pixel corresponds (2618) to one ormore image sensors. In some implementations, each pixel corresponds to asingle image sensor. In some implementations, the process 2600partitions (2620) the image sensors into a plurality of pixels. In someimplementations, the process 2600 forms (2622) a respective vector ofthe received IR images for each pixel. For each pixel, the process 2600estimates (2624) a depth in the scene at the respective pixel by lookingup the respective vector in a respective lookup table. Someimplementations use lookup tables constructed as described above withrespect to the process 2200 in FIGS. 22A-22C.

A user designates (2626) a zone within the RGB image. In someimplementations, the designated zone is a region of interest, such as aregion with special monitoring. In some implementations, the specialmonitoring consists of excluding the region from monitoring movement. Insome implementations, an alert is triggered when there is movement in adesignated zone. In some implementations, the zone corresponds (2626) toa contiguous plurality of pixels. In some implementations, the zone is(2628) a quadrilateral. In some implementations, the zone is a polygon.In alternative implementations, the user designates a zone within an IRimage instead of within an RGB image.

The process 2600 receives (2630) a second plurality of distinct IRimages of the scene taken by the array of image sensors at a second timethat is after the first time. In some implementations, each of the IRimages in the second plurality is captured (2632) while a differentsubset of IR illuminators of the camera system is emitting light.Typically, the subsets of IR illuminators used to capture the secondplurality of IR images are the same as the subsets of IR illuminatorsused to capture the first plurality of illuminators.

The process 2600 then uses (2634) the second plurality of IR images toconstruct a second depth map of the scene. The process 2600 typicallyuses the same steps for building the second depth map as used forbuilding the first depth map, which was described above with respect toboxes 2618-2624 in FIG. 26A.

The process 2600 then determines (2636) physical movement of the camerasystem based on the first and second depth maps. In many cases, if therehas been no movement of the camera, the second depth map issubstantially the same as the first depth map. However, in some cases,objects in the scene itself change, such as placing a new item offurniture in the monitored area, placing new artwork on a wall, or evenaccumulated clutter on a floor.

In some instances, the determined physical movement is (2638) an angularrotation. In some implementations, the determined physical movement is(2640) a lateral displacement. For example, the camera may be bumped alittle to the left or the right on a shelf. Note that lateraldisplacement can be a horizontal movement, a vertical movement, and/or amovement forward or backward. In some implementations, a “lateraldisplacement” is defined as any movement of the camera 118 in which thecamera continues to point in the same direction (e.g., due east). Inmany cases, if the camera 118 is bumped or nudged, the physical movementincludes (2642) both an angular rotation and a lateral displacement.

In some implementations, the process 2600 identifies (2644) a pluralityof points in the first depth map and a corresponding plurality of pointsin the second depth map. The process 2600 then determines (2646) arespective displacement for each of the identified points between thefirst and second depth maps. By combining the displacements for aplurality of distinct points, the process 2600 determines the overallmovement of the camera 118.

In some implementations, determining the movement of the camera usespoint clouds. The process 2600 forms (2648) a first point cloud using afirst plurality of points from the first depth map, and forms (2650) asecond point cloud using a second plurality of points from the seconddepth map. The process then computes (2652) a minimal transformationthat aligns the first point cloud with the second point cloud. One ofskill in the art recognizes that correlating two point clouds can beperformed in various ways. Based on the point cloud transformation, theprocess 2600 identifies the motion of the camera 118 that would producethe point cloud transformation.

Based on the determined physical movement of the camera system 118, theprocess 2600 translates (2654) the zone in the first RGB image into anadjusted zone. When the zone originally designated by the user is aquadrilateral, the adjusted zone is (2656) also a quadrilateral.However, because of the transformation, in some instances, a first edgeof the quadrilateral has (2658) a length that is different from acorresponding second edge of the second quadrilateral.

In some implementations, the process 2600 receives (2660) a second RGBimage of the scene taken by the array of image sensors of the camerasystem temporally proximate to the second time. In some implementations,the process 2600 correlates (2662) the adjusted zone to a set of pixelsfrom the second RGB image. This can be helpful to a user who wants toview the zones.

FIGS. 27A-27D provide a flowchart of a process 2700, performed by acomputing device, for estimating (2702) the height and tilt angle of acamera system having a 2-dimensional array of image sensors and aplurality of IR illuminators in fixed locations relative to the array ofimage sensors. The height and tilt angle are measured with respect to afloor near the location of the camera system. The method is performed(2704) at a computing device (e.g., a scene understanding server 900)having one or more processors and memory. The memory stores (2704) oneor more programs configured for execution by the one or more processors.In some implementations, the computing device is (2706) a serverdistinct from the camera system. In other implementations, the computingdevice is (2708) included in the camera system.

The process 2700 identifies (2710) a plurality of distinct subsets ofthe IR illuminators. Subsequently, each of the distinct subsets ofilluminators are activated one subset at a time, and the images capturedwith different illumination enables determination of the camera heightand tilt angle. In some implementations, each of the distinct subsets ofthe IR illuminators comprises (2712) two adjacent IR illuminators, andthe distinct subsets of the IR illuminators are non-overlapping. In someimplementations, each individual illuminator is one of the distinctsubsets. For example, if a camera system has eight illuminators, someimplementations have eight distinct subsets, consisting of eachindividual illuminator. In some implementations there is overlap betweenthe distinct subsets. For example, in a camera system with eightilluminators, some implementations have eight distinct subsetscorresponding to each possible pair of adjacent illuminators. One ofskill in the art recognizes that many other selections of subsets of IRilluminators are possible.

The process 2700 also partitions (2714) the image sensors in the arrayinto a plurality of pixels. In some implementations, each pixelcomprises (2716) a single image sensor. In other implementations, eachpixel comprises (2718) a plurality of image sensors. Typically, theimage sensor array 852 has a large number of image sensors (e.g., amillion or more). Implementations commonly downsample the images,combining multiple sensors into a single virtual pixel. In someimplementations, each pixel includes about 100 image sensors (e.g., a10×10 contiguous square). In some implementations, each pixelcorresponds to the same number of image sensors.

Before computing an actual camera position, implementations build adictionary (also referred to as a training set). An example dictionary2150 is provided in FIG. 21C above. Typically, the dictionary isconstructed once, and used many times. The dictionary is constructedbased on characteristics of a specific camera, but there are generallymany cameras that can use the same dictionary (e.g., a million instancesof a single camera model can all use the same dictionary as long as thecameras are substantially identical). The dictionary consists of aplurality of entries, each corresponding to a (height, tilt angle) pair.The height and tilt angle represent the relationship of the camera(i.e., the image sensor array 852 of the camera) relative to a floornear where the camera is located. In some implementations, all of the(height, tilt angle) pairs are unique, but in other implementations, twoor more dictionary entries have the same height and tilt angle. In someimplementations, the dictionary entries are constructed based onsimulation (e.g., simulating a specific height and tilt angle above afloor, and simulating illumination from the identified subsets ofilluminators). In other implementations, the dictionary entries areconstructed based on experimental data (e.g., placing the camera atvarious heights and tilts and capturing images based on activating thevarious identified subsets of illuminators).

For each of a plurality of heights and tilt angles, the process 2700constructs (2720) a dictionary entry that corresponds to the camerasystem 118 having the respective height and tilt angle above a floor.The respective dictionary entry includes (2722) respective IR lightintensity values for pixels in images corresponding to activatingindividually each of the distinct subsets of the IR illuminators. Forexample, in some implementations with 15,000 pixels and four subsets ofilluminators, each dictionary entry has a light intensity value for eachof the 60,000 pixel/subset combinations plus the height and tilt angle(e.g., a vector with 60,002 entries). In some implementations, thedictionary entries only include pixels that correspond to the simulatedfloor. For example, if there are 15,000 pixels for the entire sensorarray, the simulated floor may occupy 3000 pixels, thus creatingdictionary entries with 12,002 components (12,000 componentscorresponding to the pixel/subset combinations, and two components forthe height and tilt angle). Some implementations have about 100dictionary entries (e.g., with height values of 0.0 meters, 0.3 m, 0.6m, . . . , and tilt angles of −40°, −30°, −20°, . . . ). Someimplementations include more entries to provide greater accuracy (e.g.,height values every 0.1 meter and angles every 5 degrees).

In some implementations, the constructed dictionary entries are (2723)based on simulating the camera, the floor, and the images, and computingexpected IR light intensity values for pixels in the simulated images.In some implementations, each expected IR light intensity value is(2724) based on characteristics of the IR illuminators. As notedpreviously, the characteristics may include (2724) one or more of: lux,orientation of the IR illuminators relative to the array of imagesensors, and location of the IR illuminators relative to the array ofimage sensors. In some implementations, a respective dictionary entryfor a respective height and respective tilt angle is (2725) based onmeasuring IR light intensity values of actual images captured by thecamera having the respective height and respective tilt angle withrespect to an actual floor.

In some implementations, the process 2700 normalizes (2726) each of thedictionary entries. In some implementations, this accounts for differentsurface reflectivity. In some implementations, the process 2700normalizes (2728) each dictionary entry by determining (2728) arespective total magnitude of the light intensity features in therespective dictionary entry and dividing (2728) each component of therespective dictionary entry by the respective total magnitude. Forexample, with a dictionary entry having 12,002 elements, compute thetotal magnitude of the first 12,000 entries (corresponding to lightintensity at pixels) and divide each of those 12,000 entries by thetotal magnitude. If the light intensity features are labeled x₁, x₂, . .. , x₁₂₀₀₀, then in some implementations the total magnitude is √{squareroot over (Σ_(i=1) ¹²⁰⁰⁰(x_(i))²)}.

In some implementations, the dictionary entries are constructed at acomputing device that is distinct from the camera system, thendownloaded (2730) to the camera system from the computing device duringan initialization process. In some implementations, the subsequentdetermination of height and tilt angle is calculated at the camerasystem 118, even when the building of the dictionary is performed at aseparate computing device (e.g., a scene understanding server 900).

For each of the plurality of distinct subsets of the IR illuminators,the process 2700 receives (2732) a captured IR image of a scene taken bythe array of image sensors while the respective subset of the IRilluminators are emitting IR light and the IR illuminators not in therespective subset are not emitting IR light. In some implementations,the process 2700 receives (2734) a baseline IR image of the scenecaptured by the array of image sensors while none of the IR illuminatorsare emitting IR light, and subtracts (2736) a light intensity at eachpixel of the baseline IR image from the light intensity at thecorresponding pixel of each of the other captured IR images. This canprovide a better estimate of the light intensity due to the IRilluminators.

The process uses (2738) at least one of the captured IR images toidentify a floor region corresponding to a floor in the scene. Someimplementations use the techniques illustrated above in FIGS. 20A-20Kand 25A-25B to identify a floor region. For example, in someimplementations the process 2700 constructs (2740) a depth map of thescene using the captured IR images. In some implementations, the process2700 then identifies (2742) a region bounded by depth discontinuities.This is illustrated above in FIGS. 20B-20F. In some implementations, theprocess 2700 also determines (2744) that the region is substantiallyplanar and facing upwards.

The process 2700 then forms (2746) a feature vector including pixelsfrom the captured IR images in the identified floor region. This isillustrated in FIG. 21E. Typically, the components of the feature vectorare arranged in the same order as the components of the dictionaryentries.

The process then estimates (2748) a camera height and camera tilt anglerelative to the floor by comparing (2748) the feature vector to thedictionary entries. In some implementations, the process 2700 normalizes(2750) the feature vector and the dictionary entries prior to computingthe distances.

In some implementations, the process 2700 computes (2752) a respectivedistance between the feature vector and respective dictionary entries,and selects (2756) a first dictionary entry whose corresponding computeddistance is less than the other computed distances. In someimplementations, computing the distance between a feature vector andrespective dictionary entries comprises (2754) computing a Euclideandistance that uses only vector components corresponding to pixels in theidentified floor region. This is illustrated in FIG. 21E. For example,the actual floor may have some objects on it, such as furniture or toys.The floor identification process typically excludes these objectsbecause they are not part of the planar surface. Because the process2700 determines the height and angle relative to the floor, only pixelsthat correspond to the floor region are relevant. The process 2700estimates (2758) the camera height and tilt angle to be the height andtilt angle associated with the first dictionary entry.

Some implementations expand or modify this basic process in variousways. In some implementations, the process 2700 identifies a ceilingrather than a floor, and measures the “height” and tilt angle relativeto the ceiling. As noted above in FIGS. 20A-20K and 25A-25B, theprocesses described with respect to a floor can be used for a ceiling aswell. In this case, the dictionary entries are constructed relative tothe ceiling. In some implementations, the position of the camera iscomputed both with respect to a floor and with respect to a ceiling. Aside effect of this dual calculation is to estimate the height of theroom where the camera is located.

As noted above, the data for the dictionary entries can be constructedby simulation or by experiments with an actual camera. When formed byexperimentation, some implementations capture a baseline image for eachcamera position, and subtract the baseline from the other capturedimages with each of the subsets of illuminators activated.Alternatively, the experiments are performed in a room with no ambientlight so that each captured image represents only light comingoriginally from the activated illuminators. The size of the dictionarycan be selected based on the desired accuracy.

In some instances, multiple “floor” regions are identified. In some ofthese instances, the multiple regions are different portions of the samefloor. In other instances, one or more of the regions may be tables andone or more regions may be an actual floor. Some implementationsestimate the height and tilt angle based on each of the identifiedregions, then compare the multiple results. If they are allapproximately the same, some implementations estimate the height andtilt based on all of them (e.g., by averaging the values, taking thevalues associated with the largest region, or choosing the first one).When the heights are substantially different, some implementations takethe larger estimate, guessing that the smaller height estimate is basedon a table or other planar object above the floor. Note that the processis only an estimate. If the camera is sitting on a table and the flooris not in the field of vision of the camera, the estimated height willbe the height above the table.

Some implementations use interpolation to provide a finer estimate. Forexample, in some instances the feature vector has equally smalldistances from two dictionary entries. In some implementations, theestimated height and tilt angle are based on averaging these two closestentries. In some implementations, finding the matching dictionary entryuses a nearest neighbor algorithm. In some implementations, only thesingle nearest neighbor is used. In some implementations, the k nearestneighbors are used for a fixed small positive integer k, and a weightedaverage of these neighbors is used to compute the height and tilt angleof the camera. For example, in some implementations, the k nearestentries are selected, and each is weighted based on the inverse of itsdistance from the feature vector.

FIG. 28 provides an overview of some of the processes described herein,which utilize control of individual illuminators (e.g., LEDs) from avideo monitoring camera to collect and calculate useful information. Notshown in the overview is preliminary processing that is typicallyperformed at a server, such as building lookup tables (e.g., asillustrated in FIGS. 13, 14, 15A, 15B, and 22A-22C) or constructing adictionary (e.g., as illustrated in FIGS. 21A-21C and 27A-27D).

In the data acquisition phase 2802, the camera 118 captures (2806) IRimages while controlling which IR illuminators are on. In someimplementations, the images are captured at night, and may occurmultiple times each night (e.g., every hour). In some implementations,the camera 118 receives a command from the video server system 508 orscene understanding server 900 to collect the images. Before taking theimages, the camera typically locks auto exposure so that all of thecaptured images are taken with the same parameter settings. FIG. 14illustrates an example where the illuminators are grouped into adjacentpairs. In general, an additional IR image is taken with none of theilluminators active in order to determine the ambient light.

For cameras with substantial processing power and memory, subsequentprocessing may be performed at the camera. However, the data is commonlytransmitted to a separate server for the data processing phase 2804,which commonly occurs at a video server system 508 or a sceneunderstanding server 900. In some implementations, the data istransmitted from the camera to an external computing device in a nativeformat (e.g., five IR images). In some implementations, some processingoccurs on the camera before it is transmitted. For example, in someimplementations, the images are downsampled at the camera, which reducesthe amount of data transmitted. In some implementations, the capturedbackground image is subtracted from the other images, so the datatransmitted corresponds to light from the IR illuminators, and thebackground light is already canceled out. In some implementations, thedata is transmitted as a single long array of data, such as the featurevector 2178 in FIG. 21E. In some implementations, the components of thetransmitted data are arranged differently, such as grouping together thedata for each pixel (e.g., placing a₁₁, a₂₁, a₃₁, and a₄₁ from thefeature vector 2178 together).

In some implementations, the scene understanding server 900 includes adepth mapping module 878, which computes (2808) a 3-D depth map of thescene in the field of vision of the camera. Constructing a depth map isdescribed above with respect to FIGS. 16A-16D, 17A, 17B, and 23A-23C.The depth map information is passed on to various scene understandingprocesses 2810, such as object classifiers 922, a camera pose estimator932, or a zone correction module 928. These processes compute ordetermine various information about the scene. Both the depthinformation and the scene information are passed on to the computervision engine 2812. In some implementations, the computer vision engine2812 uses the information to provide better alerts. For example, thecomputer vision engine 2812 can reduce the number of false securityalerts by excluding certain regions or by performing automatic zonecorrection when a camera is moved slightly. In some implementations,this data facilitates motion tracking and detection of humans. The dataprocessing phase is described in more detail with respect to FIG. 29.

FIG. 29 illustrates the interrelationships between some of the sceneunderstanding processes, including the inputs and outputs for each ofthe processes. The first process 2902 builds a depth map for the scenein the field of vision of the camera. The depth mapping module 878 isalso referred to as a “depth data generator,” as shown in the depthmapping process 2902. The inputs to the depth mapping process are the IRimages, as discussed above. The depth mapping module 878 creates severaloutputs, including the depth map 2912, which is also referred to as adepth image. This provides a 3D structure of the scene, as describedabove with respect to FIGS. 16A-16D, 17A, 17B, and 23A-23C. In someimplementations, the depth mapping module 878 also creates a depth edgemap 2914, which is also referred to as a depth edge image. This isillustrated above with respect to FIGS. 20B, 20C, and 20D. In someimplementations, the depth mapping module 878 computes an active IRbrightness image 2916, which represents only reflections of light fromthe active IR illuminators, and not the environmental ambient light. Insome implementations, this is performed by subtracting the baselineintensity values (when no illuminators are on) from each of the otherimages. In some implementations, the depth mapping module 878 computes asignal-to-background image 2918, which identifies the ratio of theactive brightness (from illuminator light) to the passive brightness(from the environment). When there is too much background light, it canreduce the confidence in the calculated results.

The second process 2904 identifies large planar regions, such as floors,walls, and ceilings. This process is described above with respect toFIGS. 20A-20K and 25A-25B. The floor/wall/ceiling module 926 is alsoreferred to as the planar support detection engine. Thefloor/wall/ceiling module 926 uses as inputs the depth map 2912 and thedepth edge map 2914, and identifies regions that likely correspond tofloors, walls, or ceilings. In some implementations, thefloor/wall/ceiling module 926 labels the pixels of a scene image (eitheran RGB image or an IR image) as probable floors, walls, or ceilings.This is illustrated below in FIG. 30.

The third process 2906 performs zone correction, as described above withrespect to FIGS. 19A-19I and 26A-26C. The zone correction module 928uses depth maps constructed at two different times, as well as auser-defined zone. When the zone has changed slightly, the zonecorrection module 928 recommends an updated zone, which is typicallypresented to the user for verification. In some implementations, if thecamera has moved significantly (e.g., to another room), the zonecorrection module recommends removing the zone.

The fourth process 2908 identifies specular regions in a scene, whichgenerally correspond to windows, televisions, or sliding glass doors.This process is described above with respect to FIGS. 18A-18E and24A-24C. The window detection module 924 is sometimes referred to as aspecular region identification engine. The window detection module 924uses the depth map 2912 and the active IR brightness image 2916 toidentify regions that are probable windows, and typically uses otherinformation to make such a confirmation. For example, in someimplementations, the window detection module 924 uses the size of theregion (e.g., is it too big or too small to be a likely window). In someimplementations, the window detection module 924 uses the shape of theregion based on the empirical fact that most windows are rectangular.Based on some distorting effects, an object that is rectangulargenerally appears as a quadrilateral in an image, and thus someimplementations do quadrilateral fitting for windows.

The information provided by the scene understanding server can be usedin various ways to reduce false motion alerts. For example, anidentified specular region (identified as a possible window), may be atelevision set. In some implementations, a rectangular specular regionthat includes lots of motion is identified as a probable television.When a television is identified, “movement” within the television regionthat would otherwise create a false motion alert can be avoided. In someimplementations, false motion alerts from ceilings can be avoided aswell. Typically, “motion” on a ceiling is caused by lights, such asheadlights from cars, and should not trigger a motion alert.

Some implementations are able to identify other characteristics of thecamera location as well. For example, some implementations determinewhether the camera is inside or outside (e.g., based on the presence ofa ceiling). When a camera is inside, some implementations determinewhether the room is a small room or a large room. These characteristicscan help determine when to create motion alerts. For example, when acamera is outside, there are many regions where motion would be expected(e.g., plants or trees flowing with the wind). Therefore, motiondetection may be limited to very specific areas and/or set at a highthreshold for what triggers a motion event. In some implementations, theinformation about the camera environment (e.g., floors and windows) isused to make recommendations on where to place the camera and/or torecommend zones for more detailed monitoring. For example, in FIGS. 18Dand 18E, the camera appears to be sitting on or close to a floor. Insome implementations, the system recommends placing the camera at ahigher location.

FIG. 30 illustrates conceptually how the information is provided by thescene understanding server 900 in some implementations. The image pixelsare arranged in a two dimensional grid 3002. The grid 3002 includes manyindividual grid cells 3006, such as the grid cell 3006-1. Within eachgrid cell 3006, codes are used to provide information about what isestimated to be in that cell. The legend 3004 gives some example cellcodes that use single characters. Some implementations use numericcodes, or use bit positions within an encoded number to specify what isin the cell. The grid corresponds to the selected pixels, which aretypically downsampled from the individual image sensors from the sensorarray 852. In some implementations, the pixels form a 94×162 grid. Insome implementations, the pixels are substantially square, but in otherimplementations, the pixels are rectangular, as depicted in the examplein FIG. 30 (e.g., each pixel may correspond to an 8×12 group of imagesensors).

As illustrated in FIG. 30, some of the grid cells have information thatidentifies the type of object believed to be in the cell. For example,the upper right grid cell 3006-2 is encoded with a “C” to indicate thatit is believed to be part of a ceiling. In this example, there areseveral cells in a contiguous region 3008 that are believed to be partof a ceiling. Although the region 3008 is identified in the figure,implementations typically do not store a region definition with the grid3002. Instead, the encoded individual cells, such as the cell 3006-2provide the information.

Similarly, a group of cells including the cell 3006-3 are encoded with a“W,” indicating that the cells are part of a probable window. The region3010 includes these cells. Also, on the left is a group 3012 of cellsthat include the cell 3006-4, which is identified as a probable wall. Insome implementations, an individual cell can be labeled with at mostobject type, but in other implementations, each cell can have two ormore designations. For example, the dark region 1822 in FIG. 18D appearsto be a window, but it is also part of a door. In some implementations,the designations of “door” and “window” are compatible, so both areincluded. In some implementations, when there are two or moredesignations (which are potentially incompatible), each of thedesignations has an associated probability.

Although the grid 3002 in FIG. 30 shows only probable designations ofobjects in the monitored scene, some implementations provide additionalinformation with the grid cells. For example, in some implementations,each pixel has an associated IR and/or RGB image value. In someimplementations, each grid cell 3006 includes the estimated depth fromthe computed depth map 876. In some implementations, the grid cellsencode a computed depth edge map 944 as well, such as the depth map 944in FIG. 20E. In general, whatever features are computed for individualpixels are stored in the grid 3002.

Some implementations provide zone correction, as illustrated in FIGS.19A-19I and 26A-26C above. Some implementations address camera movementmore generally, recognizing that there are both small moves (e.g., abump) and large moves (e.g., taking the camera to a different room). Ina small move, the camera sees substantially the same field of vision, asillustrated in FIG. 19F. In this case, an activity zone can generally beadjusted. In a large move, the camera sees a substantially differentfield of vision. The previously defined activity zone is now irrelevantto the current field of vision, so the zone should be discarded. Whethera small move or a large move, some implementations issue a camera movealert so that the user can take appropriate action. Some implementationsuse push notifications to alert the user of a camera move event, butother implementations use pull notifications, allowing the user toreceive a camera move event only when requested. Some implementationssupport both push and pull notifications, and select the type based onthe importance. For example, some implementations use push notificationswhen there is a detected motion event (e.g., a possible intruder), butuse pull notifications for camera move events. Some implementationstrack the history of camera move events, and provide the user withaccess to that history. In some implementations, each camera move eventhas additional data that is stored. For example, some implementationsstore the model of the camera, the software or firmware version, theexisting activity zones, an identifier for the camera when a householdhas more than one camera, one or more timestamps to indicate when thecamera moved, the recommended action, and so on.

FIGS. 31A and 31B illustrate a camera that has moved slightly. Betweenthe time of the image in FIG. 31A and the time of the image in FIG. 31B,the field of vision of the camera appears to have moved a little to theright and a little up. Using the techniques described above in FIGS.19A-19I and 26A-26C, a recommended zone correction is determined. Analert or notification is then sent to the user, as illustrated in FIGS.31C and 31D. In some instances, the notification is sent as an email. Asindicated in FIG. 31C, the email message body indicates that the camerahas moved, and identifies the zone. In this example, the zone has beenpreviously labeled “Doorway From Kitchen” by the user. The notificationmessage also includes the image in FIG. 31D. Superimposed on the imageare the current zone 3102 (solid outline) and the recommended adjustedzone 3104 (dashed outline). In some implementations, the zones areoutlined in color to make them more visible, using a color such as neongreen. The message makes it easy for the user to accept the recommendedzone adjustment (e.g., by clicking a link or button in the message).

FIG. 31E illustrates a large move. Previously, the zone 3120 wasidentified as the “Garage Door,” whereas it now appears to be in afamily room or office. Using point cloud registration as described abovewith respect to FIGS. 19A-19I, the zone correction module 928 determinesthat the current point cloud is not a transformed version of theprevious point cloud (the one having the garage door). Therefore, anotification message 3124 is sent to the user (e.g., by email, textmessage, or instant message). The message 3124 concisely points out theissue, and provides a simple way for the user to resolve the problem(e.g., delete the zone). The message 3124 also includes an image 3122representing the current field of vision of the camera with the currentzone 3120 identified. In this way the user can easily see the problemand resolve it quickly. If the user wants to create one or morereplacement zones, the user can go into the application and create newzones.

In situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about a user'ssocial network, social actions or activities, profession, a user'spreferences, or a user's current location), or to control whether and/orhow to receive content from the content server that may be more relevantto the user. In addition, certain data may be treated in one or moreways before it is stored or used, so that Personally IdentifiableInformation (“PII”) is removed. For example, a user's identity may betreated so that no PII can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby a content server.

It is to be appreciated that one or more implementations disclosedhereinabove is particularly advantageous for application in the homemonitoring context, for which there are particular combinations ofdesirable goals including low cost hardware, very low device power(especially for battery-only devices), low device heating, nonintrusivedevice operation, ease of device installation and configuration,tolerance to intermittent network connectivity, low-maintenance ormaintenance-free device operation, long device lifetimes, the ability tooperate in a variety of different lighting conditions, and so forth, thehome monitoring context further involving particular sets of expectedtarget characteristics and/or constraints for which the preferredimplementations may be particularly effective, such as the statisticallyprominent presence of certain target types (humans, pets, houseplants,ceilings, floors, furniture, doors, windows, household fixtures, varioushousehold items, etc.), the fact that the monitoring device is usuallystationary relative to the monitored space, the fact that certain targettypes have certain expected ranges of sizes and characteristics (e.g.,humans and pets have certain sizes and any movement is usually parallelto a floor or stairway; floors-ceilings-walls are also usually ofcertain size or height ranges and are stationary; doors-windows rotateor slide within expected ranges; furniture is usually stationary and hascertain expected sizes), and so forth. However, it is to be appreciatedthat the scope of the present teachings is not so limited, with otherimplementations being applicable for the monitoring of other types ofstructures (e.g., multi-unit apartment buildings, hotels, retail stores,office buildings, industrial buildings) and/or to the monitoring of anyother indoor or outdoor facility or space. It is to be still furtherappreciated that, while facility or space monitoring represents oneparticular advantageous application, the scope of the present teachingscan further be applicable to any field in which automated machinecharacterizations of stationary or moving objects, facilities,environments, persons, animals, or vessels, are desired based onoptical, ultraviolet, or infrared electromagnetic reflection or emissioncharacteristics.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first userinterface could be termed a second user interface, and, similarly, asecond user interface could be termed a first user interface, withoutdeparting from the scope of the various described implementations. Thefirst user interface and the second user interface are both userinterfaces, but they are not the same user interface.

The terminology used in the description of the various describedimplementations herein is for the purpose of describing particularimplementations only and is not intended to be limiting. As used in thedescription of the various described implementations and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages that are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The implementations were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the implementationswith various modifications as are suited to the particular usescontemplated.

1. A method for generating depth maps, comprising: in a camera having aplurality of illuminators, a lens assembly, an image sensing element, aprocessor, and memory, wherein the illuminators are configured tooperate in a first mode to provide illumination using all of theilluminators, the lens assembly is configured to focus incident light onthe image sensing element, the memory is configured to store one or moreprograms to control operation of the camera, for execution by theprocessor, the method comprising: reconfiguring the plurality ofilluminators to operate in a second mode wherein each of a plurality ofsubsets of the plurality of illuminators provides illumination of ascene separately, including, for each subset; activating theilluminators in the subset without activating the illuminators not inthe subset; receiving reflected illumination from the illuminated sceneincident on the lens assembly and focused onto the image sensingelement; measuring light intensity values of the received reflectedillumination at the image sensing element; and transmitting to a serverthe measured light intensity values for processing at the server
 2. Themethod of claim 1, further comprising storing to the memory the measuredlight intensity values associated with activation of the subset.
 3. Themethod of claim 1, wherein differences in the measured light intensityvalues correlate with spatial depth of an object in the scene from whichreflected light was received at the respective image sensor.
 4. Themethod of claim 1, further comprising: storing to the memory themeasured light intensity values associated with activation of thesubset; capturing a baseline image while none of the illuminators areemitting light, wherein the captured baseline image measures ambientlight intensity of the scene at each of the image sensors; storing thecaptured baseline image to the memory; and for each image sensor,subtracting the baseline intensity value from the stored intensityvalues for the respective image sensor to correct the stored intensityvalues for ambient light at the scene.
 5. The method of claim 1, whereinthe image sensors are partitioned into a plurality of pixels; and foreach pixel of the plurality of pixels: using the captured IR images toform a respective vector of light intensity at the respective pixel; andestimating a depth in the first scene at the respective pixel by lookingup the respective vector in a respective lookup table.
 6. The method ofclaim 5, wherein each respective lookup table includes a plurality ofnormalized IR light intensity vectors, each normalized light intensityvector corresponding to a respective depth in the first scene.
 7. Themethod of claim 5, wherein the respective lookup table is downloaded tothe camera from a remote server during an initialization process.
 8. Acamera system, comprising: a plurality of illuminators; a lens assembly;an image sensing element; a processor; memory; and one or more programsto control operation of the camera system, stored in the memory andconfigured for execution by the processor; wherein the illuminators areconfigured to operate in a first mode to provide illumination using allof the illuminators, the lens assembly is configured to focus incidentlight on the image sensing element, and the one or more programs areconfigured for: reconfiguring the plurality of illuminators to operatein a second mode wherein each of a plurality of subsets of the pluralityof illuminators provides illumination of a scene separately, including,for each subset; activating the illuminators in the subset withoutactivating the illuminators not in the subset; receiving reflectedillumination from the illuminated scene incident on the lens assemblyand focused onto the image sensing element; measuring light intensityvalues of the received reflected illumination at the image sensingelement; and transmitting to a server the measured light intensityvalues for processing at the server.
 9. The camera system of claim 8,wherein the image sensors are partitioned into a plurality of pixels;and for each pixel of the plurality of pixels the one or more programsare configured for: using the captured IR images to form a respectivevector of light intensity at the respective pixel; and estimating adepth in the first scene at the respective pixel by looking up therespective vector in a respective lookup table.
 10. The camera system ofclaim 9, wherein looking up the respective vector in the respectivelookup table comprises: computing an inner product of the respectivevector with records in the lookup table; and determining the depth inthe first scene at the pixel as a depth corresponding to a record in thelookup table whose inner product with the respective vector is greatestamong the computed inner products for the respective vector.
 11. Thecamera system of claim 10, wherein computing an inner product of therespective vector with records in the lookup table comprises: for eachrecord in the respective lookup table, computing an inner product of therespective vector and the respective record.
 12. The camera system ofclaim 10, wherein: the respective vector for a respective pixelcomprises a plurality of components, each of the componentscorresponding to a respective IR light intensity for the respectivepixel for a respective captured IR image; and computing the innerproduct comprises computing a dot product.
 13. A non-transitory computerreadable storage medium storing one or more programs configured forexecution by a camera system having a plurality of illuminators, a lensassembly, an image sensing element, a processor, and memory, wherein theilluminators are configured to operate in a first mode to provideillumination using all of the illuminators, the lens assembly isconfigured to focus incident light on the image sensing element, thememory is configured to store one or more programs to control operationof the camera, for execution by the processor, and the one or moreprograms comprise instructions for: reconfiguring the plurality ofilluminators to operate in a second mode wherein each of a plurality ofsubsets of the plurality of illuminators provides illumination of ascene separately, including, for each subset; activating theilluminators in the subset without activating the illuminators not inthe subset; receiving reflected illumination from the illuminated sceneincident on the lens assembly and focused onto the image sensingelement; measuring light intensity values of the received reflectedillumination at the image sensing element; and transmitting to a serverthe measured light intensity values for processing at the server. 14.The non-transitory computer readable storage medium of claim 13, whereineach of the subsets of illuminators is configured at a different anglerelative to the image sensing element.
 15. The non-transitory computerreadable storage medium of claim 13, wherein each of the subsets ofilluminators highlights a different portion of the scene.
 16. Thenon-transitory computer readable storage medium of claim 13, furthercomprising: transmitting the measured light intensity values to a depthmapping module configured to estimate spatial depths of objects in thescene based on the measured light intensity values, predeterminedillumination specifications of the illuminators, and responsespecifications of the image sensors.
 17. The non-transitory computerreadable storage medium of claim 13, wherein the illuminators are IRilluminators.
 18. The non-transitory computer readable storage medium ofclaim 13, wherein the illuminators comprise 8 IR illuminators and eachof the subsets of the illuminators comprises two adjacent IRilluminators.
 19. The non-transitory computer readable storage medium ofclaim 13, wherein the image sensing element is a 2-dimensional array ofimage sensors.
 20. The non-transitory computer readable storage mediumof claim 19, wherein the image sensors are partitioned into a pluralityof pixels; and for each pixel of the plurality of pixels: using thecaptured IR images to form a respective vector of light intensity at therespective pixel; and estimating a depth in the first scene at therespective pixel by looking up the respective vector in a respectivelookup table.